acceptodds
Under review as a conference paper at ICLR 2027

Relational Linearity: Probing Language Models Through Their Own Predictions

Abstract

We study relational linearity in language models: whether a single linear map on the context representations alone (“Yesterday, we watched a movie”) reproduces the model own next-token distribution for a fixed query (“What is the tense of theprevious sentence?”) over candidate answers (“past”, “present”, “future”). We introduce LiRe, a probe fitted by matching the logits of the prompted model, thus requiring no ground-truth annotation, and apply it at every layer of four language models on six queries of increasing abstraction. We find that relational linearity is more pronounced in larger models and tracks the abstraction a query demands: surface-level linguistic relations (language, verb tense) are already recoverable in early layers, whereas more abstract ones (subjectivity, faithfulness) peak in middle layers. It is also tied to how the query is formulated: e.g., translating it, affects smaller models markedly and larger ones only mildly. Finally, relational linearity can be high even when the model most likely candidate answer disagrees with the ground-truth label, highlighting a structure that would be missed if probed only with ground-truth annotations.

Primary area:
interpretability and explainable AI
Keywords:
relational linearity, representations, language models, probing, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_669DOpenReview ↗
Rating
2
Confidence
4
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed to use softmax-linear probes to detect whether or not the next-token probability distributions (and their simplified forms) can be recovered using model's residual stream vectors when fed with the sentences themselves as user prompts. They get a series of findings from their framework, like the language model's wrong answers can also be probed, and they also studied the layer-wise differences and perturbation behaviours.

Critical strengths
  1. The topic of probing is a good choice: even though probing may seem a deprecated as mechanistic methods predominate nowadays, it's still important and worth exploring, especially in activation/vector-based explainability (which I believe is necessarily complementary to mechanistic/circuit approaches). So that field still needs active research.

  2. The probability simplex flag is a valuable consideration. It is usually not straightforward to think about the distribution collapse at first glance

So given the weaknesses, which might also become critical if not well resolved, considering the positive value of this research topic, I still decide to give a "weak reject".

Critical weaknesses
  1. If the system prompt, including the instruction of the task and vocabulary constraints, is applied to the prompting process used to generate the conclusions, their work can be considered as "probing the last hidden states with the original query present, but switched to a different position". That will make the impact and surprise of their findings weaker.

  2. Since the main contribution of this paper is changing the training objective of the probe from ground-truth to model predictions, there should be more statements on the significance/soundness of switching to this objective. For example: First, a concern is that, we would generally value the case where probe CAN find the desired answer while the model FAILS to predict it well, as a more valuable case, since it probably indicates some internal representations already implicitly contain the desired answer but model fail to extract it, and it's of more value for diagnosing model failures, and also gives more surprise. The authors' proposed objective, on the other hand, sits in the opposite direction. Another concern is that, since nowadays LLMs are larger, getting linguistic/factual labels often involves cheaper classifier/smaller model runs. So “no annotations” is, at least, not always the case regarding computation/efficiency. Also, the conclusions this paper gives sound unsurprising: "We find that relational linearity is more pronounced in larger models" might also be explained as an expected result of scaling/emergence. "surface-level ... already recoverable in early layers" and "abstract ones ... in middle layers" actually stay consistent with previous linguistics probing works. "translating it, affects smaller models markedly and larger ones only mildly" is also an expected outcome of the (somehow proven) robustness of larger models.

These weaknesses let me decide the paper is not a ready-to-accept work.

Questions for authors

Questions:

  1. What's its difference from a classifier trained upon a sentence embedding model (which is the LLM), while the dataset is automatically labeled (and label-smoothed) by the probed LLM?

  2. Following W2, did the authors specifically study (better case or mechanism studies) on the properties when model's predictions DO NOT agree with ground-truth (because that is the main of difference between the authors' proposed method and previous methods)?

2 comments

We thank the reviewer for the careful reading and for highlighting the value of probing and of the simplex distribution diagnostic. This diagnostic is now analyzed in the new Appendix G.

Input to the probe (W1). The query is never part of the probe's input: we probe the representation obtained by feeding the sentence alone to the model. The task instruction, candidate answer options, and cues such as "Truthfulness:" appear only in a separate forward pass, which provides the target distribution over candidate answers. The placement of the instruction therefore affects only the target distribution. Section 3.3 of the revised PDF now states this explicitly.

Significance: model's predictions vs. ground truth.

LiRe, used alongside label-trained probes, lets us separately consider whether the following can be linearly predicted from context representations: (i) a model's own output distribution; (ii) the ground-truth labels.

Distinguishing between the two lets us inspect how the model processes information internally, even when its predictions are incorrect. For example, studying Gemma3(1B) on the TENSE dataset, we found that the model often gives incorrect answers to the query (F1(GT) = 0.59), whereas the correct answers can be read linearly by a label-trained probe on the internal activations (F1(GT) above 0.95).

In that case, the agreement of LiRe with the model's answers is nonetheless high (F1(LLM) = 0.88), giving evidence that the model's answers, including the incorrect ones, are linearly predictable from the same representations.

We believe that crisply separating linear probing of the model's own predictions from that of the ground truth is an important conceptual distinction in interpretability, and our work takes a step in that direction.

Annotations. We agree that labels may often be cheap to obtain. As we highlighted above, a key conceptual contribution of LiRe is that relational linearity can be studied in contexts in which the model is wrong. In contrast, earlier work on relational linearity (Hernandez et al., 2024b) evaluated it on examples that the model answers correctly, thus neglecting failure cases.

Expected trends. We agree that the layer-wise and scale trends are consistent with prior probing work: we do not claim them as the main contribution.

The main contribution is to distinguish linear probing of the ground truth from that of the model's own predictions in a relational setting, and a first causal validation of the recovered directions (Appendix F).

Q1 (difference from a classifier on LLM-labeled data). As a procedure, the two are close. The difference is in what is tested. Because the representation and the labels come from the same model, success indicates that this model's answer is linearly recoverable from its own context representation. This is relational linearity in the sense of Definition 1.

Q2 (cases where the model disagrees with the labels). Yes. Section 4.4 studies these cases for all models, queries and layers, by comparing how well LiRe agrees with the model's answers and with the labels, and the revision adds the comparison with label-trained probes described above. In addition, Appendix F provides new evidence on LANG: intervening along directions derived from LiRe changes the language of the model's output, and some of the errors mirror the model's own confusion between related languages, such as Russian and Bulgarian (Appendix F.1).

Reviewer_669DOpenReview ↗

Response for Authors' Rebuttal Comments

I thank the authors for their replies. For W1, I think it is completely resolved. For W2, I thank the authors for providing an example to help me understand the significance of recovering the model's predictions, while a further dive into the underlying mechanisms is still not effectively provided. I think such analysis would provide important support for the authors' claims, including the idea that the example aims to illustrate. The authors also acknowledged my concerns about whether avoiding annotations can be treated as an effective advantage, and the less surprising conclusions, some of which share principles with prior probing works, and withdrew some claims. For Q1, I think this concern becomes weaker as the significance of probing for model outputs is strengthened. For Q2, I think it is only partially resolved, because the study in Section 4.4 is still not as fine-grained as a mechanistic study.

Overall, considering the remaining concerns, including the lack of further justification for probing model predictions and the withdrawal of some conclusion claims, I decide to maintain my overall rating. However, considering the additional insights I get from the clarifications of "why probing for model predictions are useful", I have increased my informativeness score.

Reviewer_GPVCOpenReview ↗
Rating
2
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The authors study relational linearity in language models. To investigate this, they propose LiRe, a linear probing method. The key point is to train it on the model predictions rather than ground-truth labels. The authors evaluate LiRe across different layers of four language models and six linguistic and semantic tasks. The results show that some relations can be recovered linearly. The patterns across layers and models are different. They find that linear probes can reproduce incorrect model predictions. Query wording also affects smaller models more strongly. Compared with the existing LRE method, LiRe achieves better prediction recovery with lower computational cost.

Critical strengths

Understanding how language models represent relational information is important for model interpretability. Research in this direction can provide useful insights into model behavior.

The proposed probing method is simple and efficient. LiRe does not require ground-truth labels and can directly use the model predictions.

The experimental design includes useful controls and covers different settings. The authors evaluate four models on six tasks, across multiple metrics. The distinction between model predictions and ground-truth labels is interesting.

The paper provides additional analyses of model behavior. The layer-wise analysis shows how prediction recovery varies across model layers. The experiments on query formulations also provide useful observations.

Critical weaknesses

The novelty is somewhat limited. Previous work has already studied relational linearity (e.g., LRE). Tuned Lens (Belrose et al., 2023) also trains linear probes to recover a model own predictions without ground-truth labels. LiRe applies linear regression to query-specific model predictions, but its main ideas are closely related to existing methods.

The motivation and interpretability benefits are not clear. The authors argue that using model predictions instead of ground truth allows them to study relational linearity independently of correctness. However, it remains unclear how this approach improves our understanding of how models represent or use relational information. Recovering model predictions does not necessarily explain how those predictions are made.

The experimental results are mixed. LiRe performs well on some relations, but not the others. The resutls are different across datasets and layers. The evidence is weaker or inconclusive for PROP and AWARE. Some results also show high output collapse or poor distribution matching.

Some findings are expected or have been reported before. The layer-wise patterns are similar to earlier probing findings. The effect of query formulation is also expected, since changing the query can change the model predictions. It also appears that the experiments change both query wording and answer tokens. The paper provides additional observations, but the new insights are limited.

Questions for authors
  • What is the main new contribution beyond training a linear probe on model predictions?
  • What new insights into model representations does LiRe provide by using model predictions?
  • How should we interpret the mixed results (e.g., across datasets and layers) to draw conclusions and insights?
4 comments

We thank you for recognizing the relevance of relational information to model interpretability, the simplicity and efficiency of LiRe, and the value of comparing model predictions with annotated labels across several models and queries.

Comparison to related work.

We agree that fitting a probe to the model's own predictions is shared with Tuned Lens. The difference is in how the probe inputs and targets are constructed. Tuned Lens uses an intermediate representation to predict the final-layer output for the same input context (with no appended query).

LiRe takes a representation computed from the context alone and predicts the model's distribution over candidate answers when it is given that context together with a fixed query. This tests whether the model's response to the query is linearly recoverable from the context representation. LRE targets a related property, but estimates the map from a Jacobian approximation on a few examples. In our experiments, LiRe recovers the model answers better, at roughly three orders of magnitude lower cost (Section 4.3, Tables 2 and 3). We will discuss Tuned Lens in Appendix A.

What new insights into model representations does LiRe provide by using model predictions?

LiRe is meant to test how closely a model satisfies Definition 1. This informs us whether the answers to a specific query can be linearly read out from the context representations, and LiRe lets us inspect how the model processes information internally, even when its predictions are incorrect.

Moreover, LiRe evaluates a property that need not coincide with linear decodability of annotated labels, as illustrated by TENSE and LANG in Gemma3(1B). For a lengthier discussion on this, please refer to our replies to reviewers 6SEX and 669D.

Recovering model predictions does not necessarily explain how those predictions are made.

You rightly note that recovering predictions does not explain how they are made, and LiRe alone makes a narrower claim. Appendix F complements this analysis with a causal validation of LiRe on LANG.

We observe that, at the selected best layer and steering strength, adding directions derived from LiRe to the model hidden states increases the rate of correct translations into the target language relative to both a random direction and the unsteered model. This shows that the recovered directions can influence generation under intervention. Also, because of misalignment between model and ground-truth answers, steering might give a continuation that does not resemble the ground truth (see Appendix F.1 about Error Analysis).

We have added a paragraph at the end of Section 4.3 summarizing this experiment.

How should we interpret the mixed results?

We use LiRe as a tool to estimate how closely the model satisfies Definition 1 for a given query and layer. The aim is to determine where the linear approximation is supported by the data, so variation across queries and layers is expected. In general, we would not expect a successful detection method to always give high scores: any method with a high false-positive rate would achieve that.

We will clarify this point and how metrics should be understood at the beginning of Section 4.3.

Some findings are expected or have been reported before.

We agree that the layer-wise and scale trends are consistent with prior probing work: we do not claim them as the main contribution.

The main contribution is to distinguish linear probing of the ground truth from that of the model's own predictions in a relational setting, and a first causal validation of the recovered directions (Appendix F).

Thanks for the response. The differences between LiRe and prior methods are now clearer. I also agree that mixed probing results are not necessarily a weakness. The new steering experiments are useful and provide further evidence of LiRe's interpretability value.

However, I still have two main concerns.

LiRe combines established ideas from relational probing and prediction-based probing. While the setup differs from prior methods, the methodological contribution remains modest. Better prediction recovery than LRE does not necessarily mean better interpretability. I am still not fully convinced of the significance of the new contribution.

The steering results are positive, but they only cover language identification. They show that LiRe derived directions can influence generation. But note that they do not establish that the model normally uses these directions when making relational predictions.

The new results and revisions strengthen the paper, but these concerns remain. I am not yet fully convinced that it is now ready for acceptance. With the provided rating range being so much smaller this year, I would have to stay with the current rating. I leave that to the other reviewers and the chair.

Thank you for replying to our rebuttal.

Better prediction recovery than LRE does not necessarily mean better interpretability.

We do not claim that better prediction recovery alone implies better interpretability: our comparison with LRE instead focuses on (1) computational efficiency of the methods and (2) the scope of the analysis.

  1. LiRe can be computed at substantially lower computational cost, roughly three orders of magnitude in our experiments. In practice, for computational reasons, we could not use LRE in cases where LiRe can be straightforwardly applied: for example, we could not run LRE on Gemma3(27B). For the remaining query-model pairs, we could only use 20 contexts to estimate the LRE map, whereas LiRe uses the whole training set (see Appendix B for details).

  2. Our analysis separates relational linearity from agreement with ground truth. The LRE paper restricts its evaluation to examples where the model predicts the annotated answer correctly. We examine linear recoverability of the model’s predictions also when they are incorrect, and compare it with linear decodability of the annotated labels. The TENSE and LANG results for Gemma3(1B) show that these two properties need not coincide.

The steering results [...] do not establish that the model normally uses these directions when making relational predictions

The steering experiments demonstrate that directions learned by LiRe on the LANG dataset can influence generation. We have revised the opening of Appendix F to state this scope precisely. Could you please clarify which conclusions of the paper are unsupported within this scope?

Thanks for the response. The contributions are now much clearer, like the lower computational cost and the difference between recovering the predictions and the labels.

While these are useful contributions, my concern remains about their significance. To me, LiRe is a useful extension of the prior work, not a completely new probing method or theory. With regard to the Appendix F, I agree the experiments do support the narrower claim that LiRe derived directions can have an effect on generation. (Joshi et al., ICML 2026)

A lot of our discussion has been about the results in the appendix, which is tehnitically optional for reviewers. With the contributions now defined and the new results in place, I think the current version would need a significant reworking of its structure and presentation. The key claims and the new evidence for those claims need to be much more clearly put in the main text.

So I think a major revision is required, which may not be achievable in this review cycle.

Reviewer_6SEXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper is interested in the extent to which models represent relations linearly. This is related to a long-standing series of results which show that model behavior is often well approximated by linear maps, tracing to earlier probing work and more recently the “linear representation hypothesis”. The paper differs from prior work most clearly in that it trains the linear project from the input to the full logit distribution of the model, rather than from the input to the ground truth answer. This allows the authors to look at the presence of linear relationships even when the model gets the answer wrong. They also look at studies of the robustness of the map to paraphrases and find that the robustness is lower for smaller models.

Critical strengths

The primary novelty of the paper is that they train the linear maps from input to the model’s full logit distribution. This enables investigation of the linear representation phenomena even in the case that the model does not get the answer right, and implies that the linear relationship between inputs in outputs is perhaps stronger than would be inferred focusing on the ground truth answer as the target of the regression.

Critical weaknesses

My main concern is that the novelty is weak, or at least not too well motivated. I am still struggling to characterize the contribution – is this primarily a methodological contribution or is there meant to be a larger theoretical/conceptual contribution? There is pretty extensive work on the topic, most of which the authors already cite. Its well documented that models represent a lot of things approximately linearly, and that these relations vary across layers from more concrete to more abstract, and that (for many mechanistic phenomena) the trends are cleaner and more robust in larger models. So my reading is that the contribution is mostly methodological, i.e., that the paper differs from that prior work in that it looks at the linear relationship between the input and the model’s own output distribution rather than the input and ground truth. In a sense, this is an interesting extension, but it is quite modest, especially as the correlation between model outputs and correct answers will be highly correlated and so we wouldn’t expect this to dramatically change the higher level findings.

Thus, I think the primary concern is just that the contribution feels quite small. I’d welcome an explanation of the larger significance, if I am missing something.

Questions for authors
  • I don’t agree with claim (b) in the intro – the presence of linearity for wrong answers is consistent with this being related to an extrinsic linguistic variable. In particular, this is likely indicating that there is something about the data distributional properties that effect linearity, regardless of the ground truth. We should not expect models to know anything about ground trust beyond corpus statistics, which is one of the fundamental problems of distributional models of language. Some possibly related papers (please also look at their bibliographies and at the work that cites that, as its also likely related): https://proceedings.iclr.cc/paper_files/paper/2025/file/b35c6f99df526a798433fabff039bd50-Paper-Conference.pdf https://arxiv.org/pdf/2510.01685

  • For the CSS score – can you give some examples of the model’s answer not depending on context? This seems like it would not happen often. Can you show some real cases in your data for different CSS scores and some intuition for what these cases are and why they happen.

  • Figure 2 is about whether the probe does a better job capturing model behavior or ground truth answer. Perhaps I’m missing but why would we ever expect the probe to reflect GT more than the model’s answer. How could this possibly be the case? Of course the internal state of the model should be most predictive of the model’s own behavior, that is just what it means to be the internal state of the model no?

2 comments

We thank you for the thoughtful review, and for inviting us to explain the larger significance.

Novelty of the work. We view our contribution as mainly methodological: we introduce an efficient method to assess relational linearity, and we test it on several combinations of datasets and models. Our LiRe probe tests whether the model's next-token distribution over the candidate answers, when it is given the context together with the query, can be reconstructed by a linear map from its representation of the context alone.

Standard linear probes, which are fitted against ground-truth labels, do not test this.

the correlation between model outputs and correct answers will be highly correlated and so we wouldn't expect this to dramatically change the higher level findings

This is a natural expectation, and it holds where the model answers correctly: for Llama3.1 and Gemma3(27B) on LANG, TENSE and TRUTH (0.87 to 0.98 F1 against the labels, Table 1), label-trained probes and LiRe give the same picture (Appendix C.8).

However, our results contradict this expectation where the model's answers and the labels differ: there, probing against labels and probing against the model's answers can lead to different conclusions. Take Gemma3(1B), which scores F1(GT) of 0.59 on TENSE and 0.37 on LANG (Table 1) when prompted with the tense and language detection queries. In both cases label-trained linear probes read the correct answer off the context representation almost perfectly (F1(GT) of about 0.98 on TENSE and 0.99 on LANG, Fig. 6). Using LiRe instead:

  • TENSE: LiRe predicts the model's answers at F1(LLM) = 0.88 (Table 3). The model is often wrong, but its mistakes are linearly predictable from the context alone.
  • LANG: LiRe predicts the model's answers at only F1(LLM) = 0.37, against 0.81 for Gemma2 and 0.97 for Llama3.1 (Table 3). From the same representation, a linear probe predicts the correct language far better than the model's own answer.

So whether a property is decodable from a representation, and whether the model's answer is a linear readout of that representation, are different questions that can lead to different answers. Standard probes test only the first; our method tests the second, and shows that the two come apart.

The TENSE comparison is now in Section 4.4.

I don't agree with claim (b) in the intro – the presence of linearity for wrong answers is consistent with this being related to an extrinsic linguistic variable

We acknowledge that the wording was misleading. We have replaced the contribution with

(b) The model's answers can be recovered by a linear probe even when their agreement with the annotated labels is limited. The same context representations can support linear recovery of both the annotated property and the model's answers, even when the two diverge (Section 4.4).

Thank you also for the two references, which we will appropriately cite and discuss in Appendix A.

[...] why would we ever expect the probe to reflect GT more than the model's answer. How could this possibly be the case?

We agree that a positive ΔF1 is expected because LiRe is fitted to the model's outputs. The plots indicate that, for high F1(LLM) and large ΔF1, the probe recovers the model's answers to a specific query while agreeing much less with the ground-truth answers to that query.

We hope that this, with the inclusion of the analysis from Fig. 6 in Section 4.4, clarifies how LiRe and standard probing differ.

CSS examples.

We introduced the score in order to distinguish cases where relational linearity holds with a "trivial" probe. For example, suppose that a model assigns the same answer distribution to all contexts. The model's own logits can be predicted trivially by just setting the weights of our linear probe (Equation 5) to zero and keeping only the biases: this would lead to a high relational linearity.

The CSS allows us to distinguish cases like this from others where relational linearity holds when the model's prediction depends on the context.

To build intuition, consider the following examples:

  1. When a model selects the same next token for all contexts with full confidence, CSS=1. In the revised manuscript we uploaded to OpenReview, we prove a more general result (Appendix G): CSS=1 if and only if the model assigns the same probability distribution over the candidate answers to every context.

  2. CSS=0 if and only if the model spreads the probability mass uniformly on the vertices of the simplex: this corresponds to a model that has high confidence over its answers, but these answers change in a context-dependent way.

  3. CSS can also be high when the model's probabilities over the candidate answers vary across contexts but stay uncertain, far from the vertices of the simplex. This is better understood when looking at the connection between the CSS and the KL divergence between the model's own distribution and LiRe with weights set to zero, which we describe in Appendix G.

We find instances of all three cases in our experiments (Figs. 13 and 15 in Appendix G). Gemma3(1B) on AWARE is close to case 1: it answers "Real" with probability close to one for most transcripts (CSS= 0.777). Gemma3(27B) on TRUTH is close to case 2: it answers with high confidence on almost every sentence, and its answers are split between True and False (CSS= 0.023). Llama3.1 on AWARE is close to case 3: its probabilities spread over a broad range of uncertain values (CSS= 0.796).

Of all these cases, the one where relational linearity is intuitively most interesting is case 2, which the CSS correctly flags with a low value: there a linear probe can match the model's answers only by using the context representation. In cases 1 and 3, a constant prediction is already close to the model's answer distributions. Such cases do occur in our data: 4 of the 24 model-query pairs have CSS above 0.75, all on PROP and AWARE, and we interpret their probing results with caution (Table 1).

Prompted by your request, the new Appendix G analyzes the score across our queries. We also propose to rename it to Context-Independence Score (CIS) throughout the paper, without changing its definition: as case 3 shows, high values need not arise only from confident predictions concentrated on a single answer.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers

We thank the reviewers for their positive comments about our work: that it is an interesting extension of the mechanistic interpretability literature on probing (6SEX, 669D, GPVC), and that LiRe is both a simple and efficient method (GPVC). We are also glad that reviewers found our experiments extensive (*GPVC) and that our flag on simplex distributions is useful (669D).

The reviews also prompt us to better frame our contribution and its novelty with respect to previous works. Whereas most previous works focus on probing ground-truth answers from learned representations, LiRe lets us consider whether a model's own output distribution can be linearly predicted from context representations for textual inputs with a relational structure. Our analysis uncovers that linear probing of a model's own distribution can lead to different conclusions than probing for ground-truth labels, as we explain in the response to reviewer 6SEX. We can thus study how the model processes information internally, even when its predictions are incorrect (see response to reviewer 669D).

To complement our initial analysis, the new Appendix F (in the revised PDF, and summarized at the end of Section 4.3) contains a causal validation of LiRe showing that the recovered directions can influence generation under intervention, thus addressing a comment by reviewer GPVC. The new Appendix G analyzes the simplex diagnostic. All changes are marked in red in the revised PDF.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewers,

Thank you for taking the time to evaluate our work. As the discussion period draws to a close, we would appreciate an update on your assessment in light of our responses, manuscript revisions, and additional experiments. We believe these address the main concerns raised in the reviews.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.