- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether transformers predicting interactions between stochastic agents represent each agent’s hidden state separately. It shows that Bayesian beliefs can remain factorized under fully observed feedback, enabling a compact representation. Experiments on small transformers trained on synthetic sequences find that these beliefs are linearly decodable and can be selectively edited to influence predictions. Hiding part of the interaction can disrupt the factorization and increase representation dimension.
Critical strengths
- The paper combines rigorous mathematical derivations with controlled experiments, including probing and causal interventions, to support its claims about factored belief representations.
- The modeling of interacting agents through feedback loops is interesting and provides a clear framework for studying how transformers track multiple coupled processes.
Critical weaknesses
- While belief factorization is theoretically interesting, the paper does not establish how it improves the prediction, coordination, or interpretability of real LLM-based agents.
- The experiments use small transformers trained from scratch on synthetic state-machine interactions. Whether similar factored representations emerge in pretrained LLMs during realistic agent interactions remains untested.
Questions for authors
- Can the authors design experiments on pretrained LLMs to test whether agent-specific belief representations emerge and can be selectively edited during realistic interactions?
- How could belief factorization improve the prediction, coordination, or interpretability of LLM-based agents? Can the authors demonstrate a concrete practical benefit?
3 comments
Addressing Weakness 2 and Question 2 (practical benefit)
We demonstrate concrete benefits of factorized representations in controlled settings, one of them in a pretrained LLM. Whether they carry over to LLM-based agents interacting in natural language is a hypothesis that our framework makes testable.
Prediction. When a process is generated by agents in a fully observed feedback loop, a predictor can track each agent's belief separately, so the dimensions it needs grow linearly instead of exponentially with the number of agents. In our telephone process with five agents the joint belief has 59,048 dimensions, far more than either model's width, yet a transformer of width 120 predicts it as well as one of width 6,656 (losses within nats; App. J, Fig. 15a).
Interpretability and coordination. A factored representation allows an auditor to analyze and steer each agent separately. In our trained transformers, editing one agent's belief closes about 98% of the gap to the Bayes prediction (Sec. 3.3, Fig. 3). In a pretrained LLM predicting a feedback loop in context, per-agent edits at the last layer close 92.6% and 70.5%, against 84.3% and 40.0% for edits through the joint belief (Sec. 5, Fig. 5). When part of the communication between agents is hidden, their beliefs entangle: the belief dimension rises from 8 to about 22 (Sec. 4.3), and per-agent edits close 68% instead of 99% of the gap (App. L, Fig. 21). For multi-agent LLM systems, this suggests that monitoring or steering agents individually presupposes logging all their communication.
I thank the author for the newly added experiments and thus I increase the score
Addressing Weakness 1 and Question 1 (Experiments on pre-trained LLMs, see general response i)
Inspired by your suggestion, we analyzed whether similar factored representations of feedback loops can arise when evaluating pre-trained language models in-context, following the methodology of the recently published work by Balcells et al. 2026. When evaluating Qwen3.5-9B in context on long token sequences from a feedback loop between transducers of the mess3 and Wing families, results show that the LLM predicts the next tokens about as well as a Bayesian predictor that sees the last two to three tokens (Fig. 5). We observe that the Bayesian beliefs can be extracted linearly from the residual stream at the last layer with held-out R2 of 0.99 (mess3) and 0.68 (Wing), respectively (App. N). Finally, we could verify that these factored representations are causally efficacious: on the last layer, patching the belief of one agent along encoders fitted to the per-agent beliefs closes 92.6% (mess3) and 70.5% (Wing) of the null-to-target gap, i.e. of the KL distance between a write with no effect and the Bayes prediction for the patched beliefs. An encoder of the joint belief of both agents closes less (84.3% and 40.0%), on all ten sequences, indicating that the additional residual-stream structure it fits is not beneficial to next-token prediction. This is consistent with the LLM reading each agent's belief from its own directions, as the transformers trained from scratch do. The results are summarized in the new Sec. 5, including Fig. 5, with methodological details in the new App. N. Note that as an early investigation into in-context representations of feedback loops, the results remain limited to one example pair of processes and one LLM.
Nevertheless, these results still use artificial token sequences from simple transducer processes. Extending this analysis to actual language tasks remains a promising direction for future research.