- Rating
- 2
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors proposed to use softmax-linear probes to detect whether or not the next-token probability distributions (and their simplified forms) can be recovered using model's residual stream vectors when fed with the sentences themselves as user prompts. They get a series of findings from their framework, like the language model's wrong answers can also be probed, and they also studied the layer-wise differences and perturbation behaviours.
Critical strengths
-
The topic of probing is a good choice: even though probing may seem a deprecated as mechanistic methods predominate nowadays, it's still important and worth exploring, especially in activation/vector-based explainability (which I believe is necessarily complementary to mechanistic/circuit approaches). So that field still needs active research.
-
The probability simplex flag is a valuable consideration. It is usually not straightforward to think about the distribution collapse at first glance
So given the weaknesses, which might also become critical if not well resolved, considering the positive value of this research topic, I still decide to give a "weak reject".
Critical weaknesses
-
If the system prompt, including the instruction of the task and vocabulary constraints, is applied to the prompting process used to generate the conclusions, their work can be considered as "probing the last hidden states with the original query present, but switched to a different position". That will make the impact and surprise of their findings weaker.
-
Since the main contribution of this paper is changing the training objective of the probe from ground-truth to model predictions, there should be more statements on the significance/soundness of switching to this objective. For example: First, a concern is that, we would generally value the case where probe CAN find the desired answer while the model FAILS to predict it well, as a more valuable case, since it probably indicates some internal representations already implicitly contain the desired answer but model fail to extract it, and it's of more value for diagnosing model failures, and also gives more surprise. The authors' proposed objective, on the other hand, sits in the opposite direction. Another concern is that, since nowadays LLMs are larger, getting linguistic/factual labels often involves cheaper classifier/smaller model runs. So “no annotations” is, at least, not always the case regarding computation/efficiency. Also, the conclusions this paper gives sound unsurprising: "We find that relational linearity is more pronounced in larger models" might also be explained as an expected result of scaling/emergence. "surface-level ... already recoverable in early layers" and "abstract ones ... in middle layers" actually stay consistent with previous linguistics probing works. "translating it, affects smaller models markedly and larger ones only mildly" is also an expected outcome of the (somehow proven) robustness of larger models.
These weaknesses let me decide the paper is not a ready-to-accept work.
Questions for authors
Questions:
-
What's its difference from a classifier trained upon a sentence embedding model (which is the LLM), while the dataset is automatically labeled (and label-smoothed) by the probed LLM?
-
Following W2, did the authors specifically study (better case or mechanism studies) on the properties when model's predictions DO NOT agree with ground-truth (because that is the main of difference between the authors' proposed method and previous methods)?
2 comments
Response for Authors' Rebuttal Comments
I thank the authors for their replies. For W1, I think it is completely resolved. For W2, I thank the authors for providing an example to help me understand the significance of recovering the model's predictions, while a further dive into the underlying mechanisms is still not effectively provided. I think such analysis would provide important support for the authors' claims, including the idea that the example aims to illustrate. The authors also acknowledged my concerns about whether avoiding annotations can be treated as an effective advantage, and the less surprising conclusions, some of which share principles with prior probing works, and withdrew some claims. For Q1, I think this concern becomes weaker as the significance of probing for model outputs is strengthened. For Q2, I think it is only partially resolved, because the study in Section 4.4 is still not as fine-grained as a mechanistic study.
Overall, considering the remaining concerns, including the lack of further justification for probing model predictions and the withdrawal of some conclusion claims, I decide to maintain my overall rating. However, considering the additional insights I get from the clarifications of "why probing for model predictions are useful", I have increased my informativeness score.
We thank the reviewer for the careful reading and for highlighting the value of probing and of the simplex distribution diagnostic. This diagnostic is now analyzed in the new Appendix G.
Input to the probe (W1). The query is never part of the probe's input: we probe the representation obtained by feeding the sentence alone to the model. The task instruction, candidate answer options, and cues such as "Truthfulness:" appear only in a separate forward pass, which provides the target distribution over candidate answers. The placement of the instruction therefore affects only the target distribution. Section 3.3 of the revised PDF now states this explicitly.
Significance: model's predictions vs. ground truth.
LiRe, used alongside label-trained probes, lets us separately consider whether the following can be linearly predicted from context representations: (i) a model's own output distribution; (ii) the ground-truth labels.
Distinguishing between the two lets us inspect how the model processes information internally, even when its predictions are incorrect. For example, studying Gemma3(1B) on the TENSE dataset, we found that the model often gives incorrect answers to the query (F1(GT) = 0.59), whereas the correct answers can be read linearly by a label-trained probe on the internal activations (F1(GT) above 0.95).
In that case, the agreement of LiRe with the model's answers is nonetheless high (F1(LLM) = 0.88), giving evidence that the model's answers, including the incorrect ones, are linearly predictable from the same representations.
We believe that crisply separating linear probing of the model's own predictions from that of the ground truth is an important conceptual distinction in interpretability, and our work takes a step in that direction.
Annotations. We agree that labels may often be cheap to obtain. As we highlighted above, a key conceptual contribution of LiRe is that relational linearity can be studied in contexts in which the model is wrong. In contrast, earlier work on relational linearity (Hernandez et al., 2024b) evaluated it on examples that the model answers correctly, thus neglecting failure cases.
Expected trends. We agree that the layer-wise and scale trends are consistent with prior probing work: we do not claim them as the main contribution.
The main contribution is to distinguish linear probing of the ground truth from that of the model's own predictions in a relational setting, and a first causal validation of the recovered directions (Appendix F).
Q1 (difference from a classifier on LLM-labeled data). As a procedure, the two are close. The difference is in what is tested. Because the representation and the labels come from the same model, success indicates that this model's answer is linearly recoverable from its own context representation. This is relational linearity in the sense of Definition 1.
Q2 (cases where the model disagrees with the labels). Yes. Section 4.4 studies these cases for all models, queries and layers, by comparing how well LiRe agrees with the model's answers and with the labels, and the revision adds the comparison with label-trained probes described above. In addition, Appendix F provides new evidence on LANG: intervening along directions derived from LiRe changes the language of the model's output, and some of the errors mirror the model's own confusion between related languages, such as Russian and Bulgarian (Appendix F.1).