- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes that in-context belief updating in LLMs can be described as a trajectory through a low-dimensional “conceptual belief space.” It combines the Bayesian view of ICL with conceptual spaces, extended to distributions over those spaces. The experiments use short synthetic stories (SimpleStories) and Llama-3.1-8B-Instruct. The authors elicit per-sentence ratings for six concepts in each of three domains, and summarize each rating as the expectation over the 0–10 answer-token distribution. They then (i) embed the behavioral ratings and query-free residual-stream activations into 2D with UMAP, and compare concept-centroid distance matrices across behavior, activations, and, for Emotions, human valence–arousal ratings; (ii) show that linear probes on activations predict the ratings; and (iii) steer activations, reporting that off-target “steering entanglement” correlates with centroid distances in the behavioral manifold.
Critical strengths
- Studying how ICL beliefs evolve over freeform text, not just few-shot examples, is a well-motivated question. Connecting it to conceptual spaces from cognitive science is an interesting idea.
- Several types of evidence point in the same direction: the model’s ratings, its internal activations, human emotion data, and steering experiments.
- The paper tests a concrete prediction. Steering toward one concept should also shift nearby concepts in the learned space (sadness → anger). This goes beyond showing that a concept can be decoded.
Critical weaknesses
-
Beliefs are defined as distributions, but only their means are analyzed. Sec. 2 defines belief states as distributions over a conceptual space and emphasizes uncertainty. The analysis, however, uses only the expected rating (Eq. 3), so a confident belief and an uncertain belief with the same mean become the same point.
-
The Bayesian and dynamical claims are not directly tested. The paper aims to extend the Bayesian account of ICL dynamics (44–45). However, each time step is measured independently, and the belief update itself is neither modeled nor compared to a Bayesian reference. The “smooth trajectories” claim is also only qualitative. The title and abstract may overstate what is shown.
-
The probing results lack baselines. Because ratings are elicited with a query, they may be formed at query time rather than maintained during reading. Probes on query-free activations address this (371–375), but without simple baselines (e.g., a mean predictor, or a lexical/sentiment model on the text), an RMSE of .09-.11 is hard to interpret.
Questions for authors
- How do you distinguish beliefs held while reading from judgments formed at query time? For example, how do the probes compare with the baselines in W3?
- Since the full 0–10 distribution is available, does its uncertainty (e.g., variance) behave in a Bayesian way, such as shrinking as evidence accumulates and rising at plot twists?
- Do the geometric results hold when UMAP is fit on random samples instead of max-activating examples?
(Minor comments) The list of predictions skips “(2)” (l. 256–259). There is a broken reference “App. ??” (l. 420).