acceptodds
Withdrawn from ICLR 2027

Capsule Lens: Locating and Tracking Concept Geometry in Model Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding how concepts are encoded in the internal representations of machine learning models is a central problem in mechanistic interpretability, with implications for both the scientific understanding of deep learning and the trustworthy deployment of increasingly capable models. Existing approaches to interpret model representations mainly map representations onto more interpretable spaces and do not directly characterize how concepts occupy representation space; various hypotheses have been proposed, but often lack of rigorous validation and largely focus on static representations. In this work, we introduce Capsule Lens, a framework that matches the region a concept occupies with a simple, trackable geometric form, a capsule, defined by several interpretable parameters, fitted in closed form to each concept's geometry and validated on held-out samples. We apply Capsule Lens in two major settings: static and dynamic representations. On static representations, we demonstrate how to locate concept geometry across various models, and how the span and norm curves uncover important geometric characteristics. On dynamic representations, we present three case studies tracking representation drifts induced by distinct training settings, CLIP pretraining, RL post-training on visual question answering, and RL post-training on mathematical reasoning. These analyses reveal qualitatively different geometric dynamics, ranging from broad network-wide restructuring in CLIP pretraining to localized and concept-specific changes in RL post-training. Our results include findings aligned with existing literature as well as novel observations. We believe Capsule Lens stands as a promising tool for locating, analyzing, and tracking concept geometry in both static and dynamic representations.

Primary area:
learning theory
Keywords:
Neural Geometry, Learning Mechanics, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UwrWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a simple method for measuring the geometry of a concept in a given model's representation space. It targets both VLMs and LLMs. For a given model (say a VLM), a dataset and a concept c (for VLMS, concepts come from Visual Genome), they first select positive (image contains c) and negative images (image does not contain c). Then, they compute simple statistics of the model representations for the positive set. This is called "locating the concept geometry". THis geometry is defined by the normalized mean of the unit-normalized samples; the largest cosine distance from any sample to that axis; and, the smallest and largest sample norms. They can also track these statistics during training. They also have a disentanglement score which measures a kind of coverage between positive space and negative space. They report results on various CLIP, Llava and Qwen models.

Critical strengths

The paper tackles an important problem: understanding how concepts are encoded in the interal representations of models.

Proposed method is simple and experiments have breadth (CLIP, Qwen, Llava; visual concepts, text concepts, etc.)

Critical weaknesses

The paper does not compare any of its results with any existing method. It neither has a sensible baseline. Therefore, it is not possible to assess whether this paper makes any progress over existing methods.

The paper draws inspiration from "diverse geometric views of concept representations (Engels et al., 2025; Modell et al., 2025; Sorscher et al., 2022), ..." However, it does not compare any results with them. Each of these references is a potential competitor. Especially Sorscher et al., since they fit an ellipsoid per concept.

Suppose there is no existing method to compare against (which is not the case), then there should be sensible baseline. For example, a random-concept control, where capsules are fit to random samples of the same size drawn from the whole dataset, and compute disentanglement against random negatives.

The comparison with cones, balls and boxes in Section 3.3 only ablates the authors’ own construction; it is not a comparison with existing methods or a control.

About negative images: the paper only says negatives are “examples that do not belong to concept c” . It doesn’t say how they are sampled, but this is crucial detail missing. For example, whether you choose hard negatives would have a drastic effect on disentanglement.

Questions for authors

How were the negatives sampled?

What does the random-concept control give?

How do capsules compare with an ellipsoid or a linear probe at the same coverage?

I should note incorporating the answers of these questions to the paper is more than a major revision.

Reviewer_4yw6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a geometric structure (capsule) to represent concepts in representation of deep networks. Rather than presenting a concept as a cluster center (e.g. in k-means clustering), or a direction (e.g. in SAEs), the paper proposes to represent each concept as a Capsule, with a unit vector as axis, a span and norm bounds. It then studies how do these capsules look and how well they cover the positive points (points with concept annotated as positive) across vision models and language models, as well as across the same model over span of pre-training or post-training.

Critical strengths

s1) The paper proposes to encode concepts in a different way than what is usually done (directions in embedding space, cluster centers, neighborhoods, etc.) and instead proposes to encode them as a capsule, which could in principle be considered as novel.

s2) Doing analysis of what happens to how concepts are encoded as pre/post-training progresses is interesting. I have of course seen papers doing analysis over the time axis, but haven’t seen much on concept-level representation of models at different layers and I think it’s good to have in the paper.

Critical weaknesses

w1) This is my primary concern. I did not understand, from the intro all the way to the conclusion, why would someone be interested in looking at capsules. There is no theory on why a capsule has the right parametrization to encode concepts. At least for the linear-representation hypothesis (which in itself has many issues), there were some toy experiments showing that the behaviour of concepts seem to be close to linear. But here while reading the text, capsules seem to come out of nowhere. The only hint at a motivation that I found was in line 140: "so that studying the concept geometry reduces to reading and tracking the fitted parameters". I don't however see this as a strong arguments, as any other parametrization could also be tracked by tracking the fitted parameters (e.g. for a cluster one can see how the cluster is moving and how many members does it have across time).

I was hoping to later see some gains of this representation in Table 3, which however doesn’t seem to be the case and the numbers in terms of coverage and Disentanglement are quite comparable across representations.

I am of course more than happy to go beyond existing frameworks of SAEs and clusters etc., but there has to be at least some sort of proof of concept or a motivating example why one should be looking into capsules.

w2) Despite not being sold on the idea of capsules (w1), I still tried to look for some evidence / finding that seems interesting based on this sort of representation (although one would still need to show that the finding, whatever it may be, is not apparent under different representations (e.g. cones)). For this I looked at Figs 2 - 6, and across none of them unfortunately I saw any finding that is interesting to me as a reader.

It seems very intuitive to me that how well a concept is represented changes as the training progresses, and the numbers are different across layers. I think the missing part is making use of these readings. For instance, if a capsule shows promise on layer X for a particular concept, does that agree with how successful the steering is at that position? Essentially, can I rely on capsule coverage and its parametrization for better interpretability/steering/any other downstream use? This is clearly missing to me, which further goes to the point made above: why should one be looking at capsules as opposed to other representations seems to not be motivated in the current draft.

I thank the authors for their submissions. While I'm happy to consider capsules as a new parametrization to encode concepts and study them, I currently don't see a motivating case on why this is the right way (w1) or results of what one can achieve with this (w2).

Happy to hear what others think!

Questions for authors

None.

Reviewer_N6gDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Capsule Lens, a framework for describing and tracking the geometry of predefined concepts in model representations. A concept is defined through positive examples, and its representations are summarized by an angular region around a mean direction intersected with a norm interval. The capsule is fitted in closed form, while span and norm coverage curves describe the distributions within these bounds. The authors evaluate held-out coverage and concept disentanglement across visual and textual concepts, multiple model components, and layers. They then present three training case studies: CLIP pretraining, PPO post-training on visual question answering, and GRPO post-training on mathematical reasoning. The reported patterns include broad geometric changes during CLIP pretraining and more localized or concept-specific changes during post-training.

Critical strengths
  • The paper is generally well written and easy to follow. The motivation is clear: a compact geometric description that can be tracked across layers and checkpoints is potentially useful for understanding representation changes.

  • The method is simple and easy to implement. The closed-form fitting procedure and the separation between angular spread and representation magnitude make the resulting measurements relatively transparent.

  • The validation covers a substantial range of concepts, models, and components, including vision encoders, multimodal components, and language models. Evaluating held-out coverage together with negative-sample overlap is a useful methodological choice, and the paper openly reports substantial differences between concept types.

  • The static and dynamic analyses provide interesting illustrations of the framework. In particular, the coverage curves contain more information than the fitted extrema alone, and the training case studies suggest useful questions about where representation changes occur.

Critical weaknesses
  1. My main concern is whether the validation establishes that capsules meaningfully locate concept-specific geometry. Table 2 reports high held-out coverage, but visual-concept disentanglement is close to the balanced baseline in many components, for example 0.504 for the Qwen2-VL ViT and 0.503 for the LLaVA projector. Under the stated balanced protocol, a score near 0.5 means that positive and negative examples enter the capsule at almost equal rates. The authors acknowledge this overlap, and concepts need not occupy exclusive regions, but high coverage alone does not establish that the fitted shape captures meaningful concept structure rather than broad properties shared by most representations. Comparisons with randomly grouped samples or a concept-independent capsule would help clarify what the validation demonstrates.

  2. The capsule and coverage curves provide a limited summary of the underlying geometry, and the paper could better establish when this summary is sufficient. A capsule summarizes angular distance from one axis and a norm interval; the two curves are marginal distributions of these quantities. They do not describe variation in directions around the axis or dependencies between angle and norm. Consequently, very different high-dimensional point clouds can produce the same summaries. Section 4.1 appropriately describes the two sharp rises as angular clusters, but the broader question remains whether these summaries retain the geometric structure needed for the paper’s interpretations. Table 3 compares mainly simple enclosing shapes, with a small disentanglement difference between capsules and cones. A comparison at matched coverage, and with a regularized low-rank ellipsoid or another approximation that retains directional variation, would better establish the value and limitations of the chosen geometry.

  3. The concept construction complicates the interpretation of the results as semantic concept geometry. Visual concepts are assigned to whole images based on the presence of a labelled bounding box, so their representations also contain other objects and scene context. Selecting images with large boxes helps, but the median box still covers only 30% of the image. Similarly, lexical concepts group token occurrences, and their stronger separation may reflect token identity rather than a broader semantic property. The inclusion of SemCor-labelled concepts is useful, but the main conclusions should distinguish these different settings. In particular, declining capsule disentanglement with depth does not establish that semantic information becomes less available; it establishes reduced separation under this particular geometric approximation.

  4. The drift measurements do not clearly separate concept-specific changes from transformations shared by the representation space. A common orthogonal rotation changes the axis-drift scores while preserving all pairwise angles, and a global rescaling changes the norm bounds without changing relative geometric relationships. Changes in the global representation mean can also affect this origin-based angular description. These are valid changes in raw activations, but interpreting them as restructuring of concept encoding requires additional controls. I would appreciate comparisons against global drift, relative relationships between concepts, and an aligned or transformation-invariant representation comparison, such as CKA [1], to establish which changes are specific to the concepts.

  5. The training case studies support narrower conclusions than a general distinction between pretraining and RL post-training. They change several factors simultaneously: architecture, dataset, training stage, optimization budget, and update method, including PPO with LoRA versus CLIP pretraining from scratch. Their different drift patterns may therefore reflect these factors as well as the objective. Moreover, the RLVR analysis tracks only 18 predefined mathematical token groups at five layers, so small changes in most tracked capsules do not establish that the model more generally remains unchanged. The case studies are useful demonstrations, but the conclusions should remain specific to these runs. A matched comparison on the same backbone and data, together with a connection between geometric changes and concept-relevant behavior, would strengthen the broader interpretation.

Questions for authors

I would appreciate it if the authors could elaborate further on the weaknesses above.

[1] Kornblith, Simon, et al. "Similarity of neural network representations revisited." International conference on machine learning. PMLR, 2019.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.