acceptodds
Under review as a conference paper at ICLR 2027

Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space

Abstract

Large Language Models (LLMs) update their behavior in context, which can be viewed as a form of Bayesian inference. However, the structure of the latent hypothesis space over which this inference operates remains unclear. In this work, we propose that LLMs assign beliefs over a low-dimensional geometric space - a conceptual belief space - and that in-context learning corresponds to a trajectory through this space as beliefs are updated over time. Using story understanding as a natural setting for dynamic belief updating, we combine behavioral and representational analyses to study these trajectories. We find that (1) belief updates are well-described as trajectories on low-dimensional, structured manifolds; (2) this structure is reflected consistently in both model behavior and internal representations and can be decoded with simple linear probes to predict behavior; and (3) interventions on these representations causally steer belief trajectories, with effects that can be predicted from the geometry of the conceptual space. Together, our results provide a geometric account of belief dynamics in LLMs, grounding Bayesian interpretations of in-context learning in structured conceptual representations.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, In-Context Learning, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xZAaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes that in-context belief updating in LLMs can be described as a trajectory through a low-dimensional “conceptual belief space.” It combines the Bayesian view of ICL with conceptual spaces, extended to distributions over those spaces. The experiments use short synthetic stories (SimpleStories) and Llama-3.1-8B-Instruct. The authors elicit per-sentence ratings for six concepts in each of three domains, and summarize each rating as the expectation over the 0–10 answer-token distribution. They then (i) embed the behavioral ratings and query-free residual-stream activations into 2D with UMAP, and compare concept-centroid distance matrices across behavior, activations, and, for Emotions, human valence–arousal ratings; (ii) show that linear probes on activations predict the ratings; and (iii) steer activations, reporting that off-target “steering entanglement” correlates with centroid distances in the behavioral manifold.

Critical strengths
  1. Studying how ICL beliefs evolve over freeform text, not just few-shot examples, is a well-motivated question. Connecting it to conceptual spaces from cognitive science is an interesting idea.
  2. Several types of evidence point in the same direction: the model’s ratings, its internal activations, human emotion data, and steering experiments.
  3. The paper tests a concrete prediction. Steering toward one concept should also shift nearby concepts in the learned space (sadness → anger). This goes beyond showing that a concept can be decoded.
Critical weaknesses
  1. Beliefs are defined as distributions, but only their means are analyzed. Sec. 2 defines belief states as distributions over a conceptual space and emphasizes uncertainty. The analysis, however, uses only the expected rating (Eq. 3), so a confident belief and an uncertain belief with the same mean become the same point.

  2. The Bayesian and dynamical claims are not directly tested. The paper aims to extend the Bayesian account of ICL dynamics (44–45). However, each time step is measured independently, and the belief update itself is neither modeled nor compared to a Bayesian reference. The “smooth trajectories” claim is also only qualitative. The title and abstract may overstate what is shown.

  3. The probing results lack baselines. Because ratings are elicited with a query, they may be formed at query time rather than maintained during reading. Probes on query-free activations address this (371–375), but without simple baselines (e.g., a mean predictor, or a lexical/sentiment model on the text), an RMSE of .09-.11 is hard to interpret.

Questions for authors
  1. How do you distinguish beliefs held while reading from judgments formed at query time? For example, how do the probes compare with the baselines in W3?
  2. Since the full 0–10 distribution is available, does its uncertainty (e.g., variance) behave in a Bayesian way, such as shrinking as evidence accumulates and rising at plot twists?
  3. Do the geometric results hold when UMAP is fit on random samples instead of max-activating examples?

(Minor comments) The list of predictions skips “(2)” (l. 256–259). There is a broken reference “App. ??” (l. 420).

Reviewer_wXKLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that LLM in-context learning can be described as a trajectory through a low-dimensional conceptual belief space. Using incremental story reading as the task, the authors elicit graded ratings of concepts after each sentence and visualise the resulting behaviour and internal activation manifolds. They show that the manifolds cluster related concepts within a conceptual domain, that this representation is linearly decodable, and that activation steering can influence belief trajectories predicted by the manifold’s geometry.

Critical strengths

The manifold and steering figures were effective at visualising the core concepts of the paper.

The authors show causal interventions that impact the behavioural and activation manifolds towards a target concept and concept cluster

Critical weaknesses

I have some concerns regarding the robustness and validity of the behavioural readout used throughout the paper. The authors prompt the LLM to rate the stories against individual concepts on a 0-10 scale, but it is unclear whether this readout reliably captures the underlying conceptual beliefs. In particular, the authors do not sufficiently establish the robustness of these scores to alternative prompt formulations, rating scales, or elicitation procedures, nor do they demonstrate that the scores constitute a valid measure of belief states.

For instance, in Fig 3, the probe prediction plots in the bottom row show independent linear predictions for individual concepts, rather than a probability distribution over concepts. This makes it unclear how accurate the linear probes are and to what extent they predict the model’s actual behaviour. The authors report 0.09, 0.09 and 0.11 RMSE for the three conceptual domains, but these values are difficult to interpret in isolation. Additionally, the authors report “strong qualitative and quantitative structure” in the Emotions and Genres domains. However, the Arbitrary domain also exhibits a statistically significant correlation in the distance matrices (r = .71, p < .001)

Questions for authors
  1. What evidence supports the interpretation of these scores as valid measures of the underlying belief states?

  2. Does the sampling procedure: “dimensionality reduction to a set of the most extreme examples of each concept, and project data points onto this space” introduce bias to these manifold clusters?

  3. How do the authors reconcile the statistically significant correlation observed in the distance matrices for the Arbitrary domain? It is also not particularly well explained why one would expect the Genres domain to be structured according to the relationships of different concepts.

Reviewer_CB7rOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes that LLMs hold beliefs over a low-dimensional conceptual belief space, and that in-context learning on free-form text can be read as a trajectory through this space as beliefs are updated. The framework extends Gärdenfors' conceptual spaces, in which concepts are regions of a geometric space, by defining belief states as probability distributions over that space and belief trajectories as sequences of these states. The authors study this in the context of story understanding. Llama-3.1-8B-Instruct reads short synthetic stories from SimpleStories one sentence at a time, and after each sentence, it is asked to rate on a 0–10 scale how much a concept applies to the story so far. The expected rating gives a behavioral belief y_t,c for each concept. Three domains of six concepts are tested. Emotions and Genres are expected to have internal structure, while an Arbitrary domain serves as a counter-example. Residual-stream activations are collected at the end of each story prefix without the rating question, and both the ratings and the activations are reduced with UMAP into a behavioral manifold (M_y) and an activation manifold (M_z). The most notable aspect is that the same conceptual structure is examined at three levels, in the model's ratings, in its internal representations, and under causal intervention. Linear probes trained on activations predict the ratings with low error. Distances between concept centroids, computed over the most extreme examples of each concept, are highly correlated between M_y and M_z, the inferred hierarchies match for Emotions and Genres, and the Emotions geometry correlates with human valence–arousal ratings. Steering with probe weights or difference-in-means (DIM) vectors shifts the model's ratings and its trajectories, and the size of unintended effects on other concepts, which the authors call steering entanglement, correlates with centroid distance in M_y. The central claim is that belief updating in LLMs follows structured, low-dimensional trajectories shared between behavior and representation, and that this geometry predicts how steering spreads across concepts. The appendices add layer-wise probe and steering analyses, a sensitivity analysis over UMAP dimensionality, and qualitative evidence that the same steering vectors change the emotional content of generated story continuations.

Critical strengths
  1. A clear framework that connects conceptual geometry to belief dynamics during reading.
    • The paper extends conceptual spaces with explicit belief states and belief trajectories, and uses the framework to study in-context updating on freeform text rather than on few-shot input–output examples.
    • Story understanding is a natural setting for this, since beliefs about a story change with each new sentence. The framework gives a shared vocabulary for describing these changes in both behavior and representations.
    • This is the paper's main contribution.
  2. Converging evidence from behavior, representations, and interventions.
    • The same concept structure is examined in elicited ratings, in activations read without the rating prompt, and under steering, and the three levels broadly agree for the structured domains.
    • Steering is tested beyond the rating task. Steered story continuations are scored by an unsteered model, which keeps the intervention separate from the evaluation.
    • The appendices test several analysis choices, including probe layer, probe calibration, persistence of steering across layers, and UMAP dimensionality.
Critical weaknesses
  1. The analyses do not test the paper's central theoretical claims.

    • Sec. 2 defines belief states as probability distributions over a conceptual space, but Eq. 3 reduces each concept to a single expected rating. Uncertainty, therefore, never enters the analysis, even though the full distribution over the eleven rating tokens is available.
    • No Bayesian model is fitted to the trajectories or compared with them, and Sec. 1 notes that modeling these dynamics directly is difficult. The abstract's claim of grounding Bayesian interpretations of in-context learning is therefore an interpretation rather than a tested result.
    • The evidence for low dimensionality concerns only the six concept centroids per domain. No variance explained or reconstruction error is reported for the trajectories, and the behavioral data is already six-dimensional by design.
    • The claim that belief updates follow smooth trajectories rests on UMAP projections of a single story (Figs. 1, 3, 5). No smoothness measure or comparison with a null is reported, and UMAP does not preserve global distances.
    • In App. K the authors shows that the activation manifold also encodes position in the story. Trajectories in M_z may therefore partly track story progress rather than belief, and the paper does not separate the two.
  2. The geometric comparisons lack controls and suitable statistics.

    • Sec. 3 describes fitting both manifolds and computing their concept centroids on max-activating examples selected by the ratings. Some correspondence between M_y and M_z centroid distances is therefore expected by construction, and no null, such as permuted concept labels, shows how much. The Arbitrary domain, intended as the counter-example, still shows r = .71 (p < .001) between M_y and M_z, and a probe RMSE of .11, compared to .09 for the structured domains.
    • Steering entanglement is not compared with a simpler predictor. Strong correlations between the raw ratings of several concept pairs is shown, and the paper does not test whether centroid distance in M_y predicts entanglement better than these raw correlations.
    • The probe RMSE of .09 is reported without a baseline. It is therefore hard to judge how much of the fit reflects the model's beliefs rather than simpler features such as story position.
  3. The scope is narrow and the counter-example results are incomplete.

    • All results come from one model, Llama-3.1-8B-Instruct with 4-bit quantization, on one synthetic story corpus, while the claims are stated for LLMs in general.
    • The evidence that steering changes generated stories is qualitative, with four continuations per condition for three stories.
Questions for authors
  1. How were the p-values for the distance-matrix correlations computed? Were they based on the 15 unique concept pairs per domain or on the full matrix, and was any permutation test used?
  2. Are the M_z centroids computed over the same max-activating examples and with the same concept labels as the M_y centroids? If so, could you report the M_y–M_z correlation after permuting the concept labels on the existing data?
  3. How is the steering effect in Fig. 6 defined? Please state the steering magnitude α, the number of stories and sentence positions it is averaged over, and the layer used for DIM steering.
  4. Sec. 5 says a linear model predicts steering effects out of sample. How is the held-out set defined? Using the existing ratings in App. G, how does centroid distance compare with raw rating correlations as a predictor of steering entanglement?
  5. Was layer 9 chosen using the same test set on which the probe RMSE is reported? Please also report a baseline RMSE on the existing data, for example, from story position alone or from the control tasks of Hewitt & Liang (2019).
  6. Figs. 10 and 11 appear to repeat Figs. 8 and 9, respectively. Could you provide the Arbitrary-domain distance matrices and steering-entanglement results that App. D describes?
  7. Three genre concepts (lighthearted, tragic, and adventure) are also style labels used to sample the training stories. Does the Genres structure hold when the existing analysis is restricted to stories from a single style?
  8. App. K shows that M_z encodes story position. How much of the M_y–M_z correspondence and of the trajectory structure remains after position is regressed out of the existing activations?
  9. Using the token probabilities already collected, does the spread of the rating distribution change over a story in a way consistent with belief updating?
  10. Please see the above Weaknesses Section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.