- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies conceptors, which are used to represent concepts within the activation of a layer of a language model. It considers two types of contrasts (difference of means) to define the contrastor:
- the bipolar contrast is built by opposing positive to negative samples, both being semanticaly opposite
- the unipolar contrast by opposing the presence of a concepts in the said samples to a set of sample without this concept
Each time, the contrast is done between two poles. The paper is interested by pooling the two poles as a weighted average, resulting into a unique mean and covariance matrix. The central theoretical results is a decomposition of the conceptor associated to this pooled covariance matrix. The authors identify three terms:
- the weighted average of both conceptors
- an adjustment of the separation between the mean of both original pole, that gives less importance to the components colinear to the main princiapl component of the within-pole data
- an adjustment of the weighted average of both conceptors to reflect how the covariance of bot pole differs
Experiments are conducted on authored and public datasets with five models.
Critical strengths
[S1] the paper studies a novel object, namely ``pooled conceptors'', and propose a derivation of its formulation to study its behavior. This last is composed of three terms that are quite intuitive to understand.
- as far as I checked, the derivation is correct
Critical weaknesses
[W1] half-convincing experiments
- most experiments are conducted with non-public and small datasets. It makes harder to appreciate the significance of the results,a at the time of review (author claim they will release data if the paper is accepted). For example, it is hard to appreciate what is data for "political stance" which may be subjective, as is a "counter-expression" for "depression". The example reported in Table 9 look like toy samples that are far from real and representative sentences. They are so "prototypical" that one wonders how they could even not lead to almost perfect separation (which is not the case in general).
- The models considered are sometimes quite old (Gemma-2, Llama 3.1) and all of them are of relatively small size for current standard, raising doubts on the generalization of the results to current and future models.
- all experiments are conducted on fixed data. There is no steering, nor even experiments with generated text. Applying the approach to e.g a classification task on public data could have been convincing as well.
- the metrics used are essentially "intrinsic", which make hard to put in perspective the claims and their consequence for real usage. For example probe loss for several datasets were estimated but the manuscript only report its correlation with the quota (section 6.4). Analysing their raw value swould have been interesting in a mechanistic interpretability perspective. Even it the correlation of strong consistent (that is far from being the case), it could mean that both metrics fail equaly.
- minor remarkss
- When available, it is better to cite the reviewed paper than the arxiv preprint. For example, (Park et al., 2023) has been published at the NeurIPS 2023 Workshop on Causal Representation Learning.
- sentence on line 156-157 has no (active) verb. Maybe "define" --> "we/let define".
- rewritting the notation and definition in appendix A.2 would help to folow the proof
- the reading of the article was very tedious. I must admit I was not very familiar with the contrastor literature but there are objective details that justify my position:
- first, reading (Postmus and Abreu, 2024) has been much easier, but it only deals with steering
- the notion of "pole" is not clearly defined. When used on line 139, it is not clear what it is. The fact that it is associated to the notion of "filter" does not help since it may be confusing with the zeros of the denominator a the transfer function of the said filter.
- and are introduced at the beggining of section 3.1 without formal definition, then not used befor the end of section 3.2; on line 140, an "attribute" is mentioned without previous introduction; in the same vein, the "weight" on lines 147 appears without clear introduction.
- at first read, it is not obvious what are and in section 6.3
- regarding how is set in all experiments, it is hard to determine. In the appendix and section 3.2, a formula is given, but it is unclear whether it is always as is
Questions for authors
[Q1] why evaluating at mid-points while deeper layer seem more appropriate to reflect concepts (and more used in mechanistic interpretability literature)?
Globaly, despite spending hours on the paper, I found it very difficult to understand. I do, however, admit that this may be down to my lack of familiarity with conceptors. However, in contrast, a paper such as (Postmus and Abreu, 2024), which deals with contrastor as well, is much easier to understand. It is true not only for the theoretical part (which is nevertheless much "lighter") but also for the experiment that rely on extrinsic metrics, which are easier to appreciate/understand.
6 comments
[Comment 2]
W4
We agree that correlations should be interpreted alongside absolute performance. Section 6.4 evaluates held-out probe ROC-AUC, and Appendix C.6, Figure 7 already presents all absolute layerwise curves across five models and six benchmarks. To make their scale explicit, the following summarizes performance at each model’s fixed midpoint layer, without selecting layers by evaluation performance:
| Benchmark | Midpoint ROC-AUC range across five models |
|---|---|
| SST-2 | 0.956–0.975 |
| Rotten Tomatoes | 0.925–0.956 |
| TweetEval sentiment | 0.953–0.980 |
| Reddit Depression | 0.978–0.991 |
| Hyperpartisan | 0.604–0.702 |
| Cross-cultural Reddit | 0.615–0.687 |
With chance ROC-AUC at 0.5, these values demonstrate substantial held-out label information, particularly for sentiment and Reddit Depression. Thus, the observed correlations are not generally associations with an uninformative probe. The weaker absolute performance on Hyperpartisan and Cross-cultural Reddit also shows why correlation magnitude must be interpreted in the context of each benchmark.
The role of quota is to characterize how retention strength relates to independently measured label decodability. We distinguish this descriptive association from claims that quota reliably selects the best layer or identifies a mechanism causally responsible for model behavior. Strong decoding and strong correlation alone would not establish those claims.
To examine behavioral consequences directly, our additional five-model generation evaluation measures changes in expressed content under conceptor interventions, using blinded A/B judgments in both presentation orders. This includes evaluation on public review texts and comparisons with prompting, ICL, and alternative activation interventions. These generated-text measurements provide complementary behavioral evidence beyond the intrinsic geometric diagnostics (please refer to W3).
W5
We would like to thank the reviewer for these detailed suggestions. We particularly appreciate this specific guidance, as making the paper clear and accessible has been a priority throughout its development. The comments help us identify where further explanation would make the manuscript easier to follow.
We will explicitly introduce “pole,” “attribute,” the activation collections, and the weighting conventions at first use, restate the necessary notation at the beginning of Appendix A.2, and clarify the symbols in Section 6.3. We will also rephrase lines 156–157, make the regularization settings explicit for each experiment, and update references to published versions where available.
Q1
We use the midpoint as a fixed, common reference for the primary geometric comparisons, keeping depth matched across methods. This choice does not assume that the midpoint is optimal for every concept.
Crucially, the main coverage finding also holds at deeper layers. Appendix C.4, Figures 5–6 evaluate every decoder layer: pooled rank-20 support achieves higher held-group coverage than the mean line in all model–concept–layer averages. At the final layer, the advantage ranges from 14.5 to 56.2 percentage points across the model–concept conditions. Comparisons against stronger source-selected single directions and the pooling-decomposition ablations also include the final block. These results show that the additional-direction coverage advantage does not depend on evaluating at the midpoint, although its magnitude varies across depth.
Thank you for the careful reading and this feedback. The paper should be accessible to readers encountering conceptors for the first time. We appreciate the comparison with [2], whose steering experiments provide a concrete behavioral motivation.
Our complementary question is what a conceptor learns when opposing example collections are pooled. Intuitively, a mean-difference vector describes one displacement, whereas a conceptor filters activation directions with different learned strengths. Our theoretical contribution separates how the pooled filter changes through the separation between the two means and differences between their covariance structures. The experiments then examine which of these components preserve contrasts in held-out contexts.
[1] Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020, July). Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 4902-4912).
[2] Abreu, S., & Postmus, J. From Steering Vectors to Conceptors and Beyond: Compositional Affine Steering Mechanisms for LLMs.
Revised Manuscript
Dear Reviewer,
Thank you again for your thoughtful feedback. We have now updated the manuscript PDF to incorporate the additional content, particularly the generated-text evaluation addressing your concern about the behavioral relevance of our findings (Section 6.7 and Appendix C.8). Additions and revised passages throughout the paper are marked in blue.
We hope these revisions address your concerns. We would welcome your assessment of whether the revised manuscript satisfactorily resolves the issues you raised and look forward to hearing your thoughts.
[Comment 1]
We sincerely thank the reviewer for the thorough, well-structured review and the carefully considered suggestions, which will help us strengthen the paper and communicate its contributions more clearly.
W1
We appreciate the concern about data access and representativeness. Our evaluation combines controlled authored collections with public natural text. The grouped geometry experiments include 1,400 public reviews across three domains and 994 public stance tweets across five topics, with an entire domain or topic excluded from fitting (Section 5.1 and Appendix B.2). Tables 11 and 13 illustrate these public examples. The main probing analysis additionally trains and evaluates classifiers on six public benchmarks, including Reddit Depression and Cross-cultural Reddit (Section 6.4 and Appendix B.3, Table 6).
Table 9 illustrates the authored BipolarConcepts collection. These controlled contrasts make the intended linguistic distinction explicit. The complementary ConceptVariants collection varies the reference while keeping the topic fixed, allowing us to examine how the fitting comparison changes the resulting geometry. This use of targeted authored examples follows the diagnostic-testing rationale of work such as CheckList [1].
For political stance, the authored collections use specified left/right policy contrasts, while the public tweet evaluation retains the original topic-specific favor/against labels. For depression expression, the authored labels describe linguistic content. We explicitly distinguish ordinary functioning, symptom absence, and flourishing and evaluate how these references affect the results (Appendix B.1, Table 10, and Appendix C.5).
Regarding near-perfect separation, the classification control on the authored depression stimuli already achieves 94.9%–99.0% held-out balanced accuracy across the midpoint models and readouts (Appendix C.3, Table 21). The main layerwise probe AUCs evaluate separate public benchmark texts. High classification accuracy can also coexist with substantial variation within each class. Our geometric measurements examine the preservation of that variation and the reconstruction of contrasts in held-out groups.
Our additional generation evaluation (see response to W3) uses 200 public Amazon, IMDb, and Yelp reviews across five models, with blinded A/B judgments in both presentation orders (see our response to W3). This extends the evaluation to behavioral effects on naturally occurring text.
W2
Our evaluation covers five instruction-tuned models across four families (Gemma-2, Qwen-2.5, Llama-3.1, and OLMo-3) spanning 2B-9B parameters and including both the 2B and 9B versions of Gemma-2 (Appendix C.1, Table 14). Across this model suite, we study three concepts through held-out evaluations, layerwise analyses, and controls for fitting counts, centering, semantic references, rank, trace, and gains. These comparisons allow us to examine how the geometric findings vary across models and experimental choices. For example, when rank, trace, and nonzero gains are matched, the pooled leading directions achieve higher held-out coverage in 251 of 270 comparisons, with an average gain of 10.64 percentage points (Appendix C.3, Table 20).
The central pooling decomposition also has a broader mathematical scope. Proposition 1 is an exact identity for full conceptors at any activation dimensionality under the stated centering and regularization assumptions (Section 4 and Appendix A.2). It therefore applies to representations from larger and newer models fitted using the same construction. We agree that extending the empirical evaluation to substantially larger models would help establish how the effect sizes change with scale. Running the full suite of held-out evaluations, layerwise analyses, and controlled ablations on 70B-class models would require computational resources beyond our current budget. Within the available resources, we prioritized systematic comparisons across five models and four families.
W3
Please refer to our response to Reviewer Bbuw’s W1. This concern was raised by all three reviewers, and we have addressed it through a generation evaluation. We provide the experimental details and results in that response.