acceptodds
Under review as a conference paper at ICLR 2027

Beliefs in feedback loops: Factored representations of interacting agents in transformers

Abstract

Complex systems, from ecosystems to brains, are usually modeled as modular components coupled through feedback rather than as indivisible wholes. Can a transformer trained on the joint behavior of interacting agents be understood in the same way? In this work, we develop a computational-mechanics framework to study agents interacting in feedback loops. We formally prove that any joint stochastic process can be presented as resulting from the behavior of agents interacting in feedback, and the resulting process is generated by a hidden Markov model whose Bayesian beliefs factorize into separate components per agent. Furthermore, our experiments show that transformers trained on such feedback sequences make use of this separability, disentangling the beliefs about the underlying factors into separate subspaces of their residual stream. This representation is found to scale linearly (instead of exponentially) with the number of agents. Interventions show that these representations are causally efficacious, allowing for targeted steering of individual agents' beliefs by selectively inserting counterfactuals. However, coarse-graining the interface is found to disrupt the factorization and can force a higher-dimensional representation despite less information. Together, the results establish both the dimensional advantage of a factorized representation and transformers' ability to exploit it, laying the groundwork for investigating traces of such compositional predictive architecture in pretrained models.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, transformers, feedback, multi-agent systems, factorized representations, belief geometry

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_of1BOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether transformers predicting interactions between stochastic agents represent each agent’s hidden state separately. It shows that Bayesian beliefs can remain factorized under fully observed feedback, enabling a compact representation. Experiments on small transformers trained on synthetic sequences find that these beliefs are linearly decodable and can be selectively edited to influence predictions. Hiding part of the interaction can disrupt the factorization and increase representation dimension.

Critical strengths
  • The paper combines rigorous mathematical derivations with controlled experiments, including probing and causal interventions, to support its claims about factored belief representations.
  • The modeling of interacting agents through feedback loops is interesting and provides a clear framework for studying how transformers track multiple coupled processes.
Critical weaknesses
  • While belief factorization is theoretically interesting, the paper does not establish how it improves the prediction, coordination, or interpretability of real LLM-based agents.
  • The experiments use small transformers trained from scratch on synthetic state-machine interactions. Whether similar factored representations emerge in pretrained LLMs during realistic agent interactions remains untested.
Questions for authors
  • Can the authors design experiments on pretrained LLMs to test whether agent-specific belief representations emerge and can be selectively edited during realistic interactions?
  • How could belief factorization improve the prediction, coordination, or interpretability of LLM-based agents? Can the authors demonstrate a concrete practical benefit?
3 comments
AuthorsOpenReview ↗

Addressing Weakness 1 and Question 1 (Experiments on pre-trained LLMs, see general response i)

Inspired by your suggestion, we analyzed whether similar factored representations of feedback loops can arise when evaluating pre-trained language models in-context, following the methodology of the recently published work by Balcells et al. 2026. When evaluating Qwen3.5-9B in context on long token sequences from a feedback loop between transducers of the mess3 and Wing families, results show that the LLM predicts the next tokens about as well as a Bayesian predictor that sees the last two to three tokens (Fig. 5). We observe that the Bayesian beliefs can be extracted linearly from the residual stream at the last layer with held-out R2R^2 of 0.99 (mess3) and 0.68 (Wing), respectively (App. N). Finally, we could verify that these factored representations are causally efficacious: on the last layer, patching the belief of one agent along encoders fitted to the per-agent beliefs closes 92.6% (mess3) and 70.5% (Wing) of the null-to-target gap, i.e. of the KL distance between a write with no effect and the Bayes prediction for the patched beliefs. An encoder of the joint belief of both agents closes less (84.3% and 40.0%), on all ten sequences, indicating that the additional residual-stream structure it fits is not beneficial to next-token prediction. This is consistent with the LLM reading each agent's belief from its own directions, as the transformers trained from scratch do. The results are summarized in the new Sec. 5, including Fig. 5, with methodological details in the new App. N. Note that as an early investigation into in-context representations of feedback loops, the results remain limited to one example pair of processes and one LLM.

Nevertheless, these results still use artificial token sequences from simple transducer processes. Extending this analysis to actual language tasks remains a promising direction for future research.

AuthorsOpenReview ↗

Addressing Weakness 2 and Question 2 (practical benefit)

We demonstrate concrete benefits of factorized representations in controlled settings, one of them in a pretrained LLM. Whether they carry over to LLM-based agents interacting in natural language is a hypothesis that our framework makes testable.

Prediction. When a process is generated by agents in a fully observed feedback loop, a predictor can track each agent's belief separately, so the dimensions it needs grow linearly instead of exponentially with the number of agents. In our telephone process with five agents the joint belief has 59,048 dimensions, far more than either model's width, yet a transformer of width 120 predicts it as well as one of width 6,656 (losses within 10−310^{-3} nats; App. J, Fig. 15a).

Interpretability and coordination. A factored representation allows an auditor to analyze and steer each agent separately. In our trained transformers, editing one agent's belief closes about 98% of the gap to the Bayes prediction (Sec. 3.3, Fig. 3). In a pretrained LLM predicting a feedback loop in context, per-agent edits at the last layer close 92.6% and 70.5%, against 84.3% and 40.0% for edits through the joint belief (Sec. 5, Fig. 5). When part of the communication between agents is hidden, their beliefs entangle: the belief dimension rises from 8 to about 22 (Sec. 4.3), and per-agent edits close 68% instead of 99% of the gap (App. L, Fig. 21). For multi-agent LLM systems, this suggests that monitoring or steering agents individually presupposes logging all their communication.

I thank the author for the newly added experiments and thus I increase the score

Reviewer_RPCSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends prior work on transformers representing factored Bayesian beliefs to agents coupled in feedback loops. Modeling agents as transducers, it proves that any joint process decomposes into a feedback loop between two agents. The loop’s belief state factorizes into per-agent beliefs, whose dimension grows linearly rather than exponentially in the number of agents. Hiding one agent’s tokens breaks this factorization and raises the required dimension. On synthetic loops built from toy HMMs, small transformers linearly encode per-agent beliefs, and these beliefs can be steered by patching. Their residual-stream rank tracks the factored dimension, and coarse-grained token streams yield higher-rank representations.

Critical strengths
  1. Clean and easy to follow formalization. Theorem 1 shows that any bivariate joint process can be written as a feedback loop between two non-anticipatory transducers. Theorem 2 shows the joint HMM operator is a tensor product. Corollary 2 shows that product beliefs remain products under Bayesian filtering, and Corollary 3 quantifies the resulting dimensional saving. Together these give the paper a solid, easy-to-follow foundation.

  2. Careful and well-controlled experiments. Models are trained close to their Bayes-optimal floors, and results are reported over multiple seeds. Probes are paired with selectivity controls (shuffled and time-shifted beliefs). The causal patching experiment goes beyond probing, with floor, null and unrelated baselines and a dose-response analysis. The authors also report failure cases (sns) and limitations openly.

Critical weaknesses

The core empirical claim is not tested where it is discriminable. Per-factor probes cannot distinguish factored from joint representations, since per-factor marginals are linear functions of the joint belief. The scaling evidence (telephone, N ≥ 3) comes from settings where the joint dimension (≥ 728 for three mess3 agents) far exceeds the residual width (120, swept only up to 240). The model may therefore be forced into a factored representation rather than preferring it. In addition, patching is done only at the final-layer residual stream, so it mainly tests the readout rather than downstream use of the beliefs.

Questions for authors

Discriminating factored vs. joint representations. Can you report effective rank in settings where the joint belief fits within the model width? Examples are the two-agent loops (joint dimension 11), the mess3↔mess3 loop (26), N = 2 telephone (80), or N = 3 with d_model > 728. Relatedly, does a linear decoder from the direct-sum belief explain the residual stream better than one from the Kronecker belief? Either result would show whether the model prefers the factored representation rather than being forced into it by limited width.

3 comments
AuthorsOpenReview ↗

Addressing Weaknesses and Questions (1/2)

Factored vs. joint probes and interventions. (see general response ii) We agree that because the Kronecker features contain the direct-sum ones, as each marginal is a linear function, per-factor probes alone cannot discriminate between factored and joint representations. Regarding your decoder ("encoder" in new terminology) question: explained variance does not favor the factored representation either, as a linear encoder of the Kronecker belief explains more of the residual stream's held-out variance than one of the direct-sum belief (R2=0.88R^2 = 0.88–0.91 vs. 0.43–0.68 in the wide telephone models; App. J). This motivates our analysis of interventions, explained below.

To differentiate whether the network causally uses structure beyond the marginal distributions, we additionally fit a joint encoder to the residual stream and used it to patch one of the agent's beliefs. We find that the joint encoder fits more of the residual stream's variance, but closes less of the null-to-target gap (95.0% for TT, 87.0% for UU) than the per-agent encoder (97.6% and 98.1%), indicating that the additional structure it captures is not beneficial for next token prediction (Sec. 3.3, Fig. 3).

To check that this test can detect a joint code, we applied it to the coarse-graining models (Sec. 4, App. M), whose Bayes belief is a product of per-agent beliefs under full observation but an entangled joint belief once UU's actions are hidden. Patching in a donor sequence's whole belief through an encoder of the agents' marginal beliefs closes 99% of the gap under full observation, but only 68% (57–85% across five replicates) with UU's actions hidden, where an encoder of the joint belief closes 99% (App. L, Fig. 21). The test thus effectively separates a factored from an entangled representation.

Effective ranks of two-agent loops and the N=2N = 2 telephone. In the small two-agent loops, the factored and Kronecker beliefs differ by only one to three components at the 80% and 95% thresholds (setup A, joint dimension 11: 2/3 vs. 3/5; mess3↔mess3, joint dimension 26: 2/4 vs. 4/7). The models' final-layer effective rank matches the Kronecker beliefs' at 80% and exceeds both at 95% (3/8–11 and 4/6–9, five seeds, on the raw residual stream and after the final LayerNorm; reported in App. J). The N=2N = 2 telephone (joint dimension 80) is similar: the models need 8–13 components at 95% (factored 5, Kronecker 8), but even the full joint belief explains only 88–93% of their activations after the final LayerNorm, so the excess is not a linear function of the belief state. In these small settings effective rank thus appears to lack the power to discriminate. For this reason, we rely on the encoder and causal tests above there (at N=2N = 2, the per-agent edit closes 98–99.5% of the gap at both widths, and the joint encoder less in every seed; App. J), and on width for the telephone task where the dimensional differences are more pronounced.

AuthorsOpenReview ↗

Addressing Weaknesses and Questions (2/2)

Wider residual streams for telephone networks (see general response iii). We furthermore agree that the models used for the telephone experiments were not wide enough to have the opportunity to build the joint belief for N>2N > 2, and thus to properly discriminate between a preference towards factored representations and its necessity given dimensional constraints. We re-ran the experiments with a residual stream width of d=6656d = 6656, which fits the tensor product belief dimension for up to four mess3 agents (94−1=65609^4-1 = 6560 dimensions). After the final LayerNorm, the wide models need 8–10 components at 95% for N=3N = 3–5, as do the factored beliefs (7–11) and the dmodel=120d_{model} = 120 models (7–10), well below the Kronecker beliefs dimensions of 19–124 (new App. J, Fig. 15d). For every N=2N = 2–5 of the wide models, the per-agent edit closes 97–99% of the gap and the joint encoder less for every seed (Fig. 16), both before and after LayerNorm.

Belief patching across depth. We agree that the final-layer patch mainly tests how the readout uses the belief encoding. We therefore repeated the belief patching of Fig. 3 (loop with TT = mess3, UU = fractal2; five seeds) at every residual-stream site of the four-layer model: the embedding, and after each block's attention and MLP sublayers (App. L, Fig. 22, solid blue). At each site the belief encoder is refit on that site's activations, and the same edit is written at the final position. Written after block 1, three blocks before the readout, the edit closes 88% (TT) and 63% (UU) of the null-to-target gap. From block 3 on it closes about 98%, as in Fig. 3. Two path controls separate the skip connections, which carry an early write unchanged to the readout, from the later blocks' response to it (App. L): written into the embedding, the edit acts entirely through the later blocks (41% for TT). We see that after block 1, the skip path alone closes 68% (TT) and 35% (UU) and the later blocks alone 50% and 44%, so the blocks after the edit turn it into the Bayes prediction for the patched beliefs. After block 2, on the other hand, the skip path carries most of the effect (92% and 85%).

Thanks for the detailed response to my questions. They mostly addressed my questions. I will raise my score then.

Reviewer_SPRLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper develops a theoretical framework for studying interacting agents in feedback loops and establishes conditions under which their joint dynamics can be factorized. Through experiments on synthetic feedback processes, the authors demonstrate that transformers can learn linearly decodable representations of individual agents’ beliefs in these settings, that these representations can be causally intervened on to steer predictions, and that factorization can reduce the dimensionality needed to represent the joint system.

Critical strengths
  1. The paper provides a useful framework for studying representations in interacting agents. It extends the study of factored belief representations to feedback settings, where agents influence one another, and examines when representing individual agents’ beliefs can reduce the dimensionality required to represent the joint system.

  2. The experiments go beyond showing that beliefs are decodable. Through counterfactual interventions, the authors show that manipulating the representations can steer model predictions toward the corresponding Bayesian predictions. This provides stronger evidence that these representations play a functional role in the model.

Critical weaknesses
  1. The relationship to prior work could be clearer. A more explicit discussion of how the theoretical results extend existing work on factored representations would help clarify the specific insights gained from studying interacting agents in feedback loops.

  2. The empirical evaluation focuses on controlled synthetic settings. While these settings allow the authors to study belief representations with known ground truth, it remains an open question whether similar representations arise in more complex systems.

Questions for authors
  1. The paper identifies discovering belief representations in pretrained models as an important next step. Beyond settings where beliefs are linearly decodable, how might the proposed probing and intervention methods be adapted to identify and causally isolate beliefs that are encoded in more distributed or nonlinear representations?

  2. The results show that transformers can learn factored belief representations of interacting agents. Could these representations be leveraged to better understand the underlying dynamics of such systems, improve inference efficiency by exploiting the factorization, or characterize how information flows between agents?

7 comments
AuthorsOpenReview ↗

Addressing Weakness 1 (relation to prior work)

The question of when a coupled system can be understood through its parts has been posed in several fields. In information theory it arose from feedback: Massey, building on Marko, noted that the joint statistics of two interacting sequences can always be split into two pieces, one for each direction of influence (Marko, 1973; Massey, ISITA 1990), and Kramer named these causally conditioned distributions and built a general calculus around them (Kramer, ETH Zurich thesis, 1998). We use this in our proof that every joint process can be interpreted as a feedback process between agents. Factorial hidden Markov models gave a clean example of a process built from independent parts whose posterior is still entangled, because the parts are seen only through a shared observation, and Ghahramani and Jordan handled this by approximating the posterior with a product of per-part beliefs (Machine Learning, 1997). Boyen and Koller then showed, for general dynamic Bayesian networks, that factored beliefs can be maintained indefinitely: provided the process mixes, the expected error they introduce stays bounded over time (UAI 1998). The same idea has recently returned in two forms: as a way to decompose world models into modular transducers (arXiv:2512.02193), and as a property of trained networks. Factored belief representations in transformers were proposed and tested by Shai et al. (2026) through the introduction of the factored world hypothesis: for a process built from independent sub-processes, or from a dependency chain whose sub-processes remain conditionally independent given the emitted tokens, the network represents the direct sum of per-factor beliefs rather than their tensor product. We extend this to agents that influence each other in both directions: in a feedback loop each agent's actions shape the other's perceptions, so neither is independent of the other, yet the Bayesian belief remains a tensor product of per-agent beliefs as long as all exchanged signals are observed (Corollary 2), and it can become entangled once part of the exchange is hidden (coarse-graining, Sec. 4). Empirically, transformers trained from scratch represent these loops in the factored, dimensionally parsimonious form, even when they have room for the joint belief (new App. J), and in a pretrained LLM evaluated in context, per-agent belief directions steer the prediction better than an encoder of the joint belief (new Sec. 5).

We added this positioning to the revised paper: the related work in the Discussion (Sec. 6.1) and the extended related work (App. A) now contrast the factored approximations of factorial HMMs with the exact factorization in fully observed feedback loops, and Kramer's causal conditioning is cited next to Massey where the decomposition is introduced (Sec. 2.2) and proved (App. G).

AuthorsOpenReview ↗

Addressing Weakness 2 (synthetic setting)

We agree that focusing our analysis to toy settings is a limitation of our original submission. Inspired by this comment, we have added an experiment on a pretrained language model (Qwen3.5-9B) predicting a feedback loop in context (see general response i; new Sec. 5 and App. N), following the methodology of the recently published work by Balcells et al. 2026. When evaluating Qwen3.5-9B in context on long token sequences from a feedback loop between transducers of the mess3 and Wing families, results show that the LLM predicts the next tokens about as well as a Bayesian predictor that sees the last two to three tokens (Fig. 5). We observe that the Bayesian beliefs can be extracted linearly from the residual stream at the last layer with held-out R2R^2 of 0.99 (mess3) and 0.68 (Wing), respectively (App. N). Finally, we could verify that these factored representations are causally efficacious: on the last layer, patching the belief of one agent along encoders fitted to the per-agent beliefs closes 92.6% (mess3) and 70.5% (Wing) of the null-to-target gap, i.e. of the KL distance between a write with no effect and the Bayes prediction for the patched beliefs. An encoder of the joint belief of both agents closes less (84.3% and 40.0%), on all ten sequences, indicating that the additional residual-stream structure it fits is not beneficial to next-token prediction. This is consistent with the LLM reading each agent's belief from its own directions, as the transformers trained from scratch do. The results are summarized in the new Sec. 5, including Fig. 5, with methodological details in the new App. N. Note that as an early investigation into in-context representations of feedback loops, the results remain limited to one example pair of processes and one LLM.

Note that this in-context analysis (covering one model and one pair of processes) still utilizes processes with known ground-truths. Going beyond known processes—towards unsupervised belief discovery in pretrained networks—is one of the outstanding long-term goals of this line of investigations, for which this initial work provides (i) the theoretical groundwork and (ii) empirical validation that transformers can express and show a preference for the representations we conjecture. Given this, searching for such representations in pre-trained models, possibly extending the sparse-autoencoder approach of Levinson (2026, arXiv:2604.02685), remains a natural path for future research.

AuthorsOpenReview ↗

Addressing Question 1 (nonlinear or distributed representations)

Our probes and interventions rest on the linear representation hypothesis of beliefs; the factored representation goes beyond it only in that the next-token probabilities of the factors multiply. The method could be adapted to nonlinear encoding by changing the regression target, i.e., by fitting the encoder to a nonlinear function of the belief, for instance the log-beliefs. The limitations section (Sec. 6.2) now states this scope, mentioning that beliefs encoded nonlinearly, for instance as log-probabilities, would escape our linear probes. Our encoders are fit to individual layers of the residual stream; for beliefs distributed across multiple layers, a single encoder can be fit on several sites jointly, with the edit written to all of them.

Comments on the paper

5 comments

We thank the reviewers for their constructive feedback. Following the received suggestions, we (i) show that LLMs factorize belief representations of feedback loops in-context, (ii) provide additional evidence of a transformer's preference for factored representations by showing that per-agent encoders beat a joint encoder in steering the transformer, and (iii) show that the effective rank of representation remains close to the factored bound even in large residual-stream transformers which could easily accommodate the non-factored, Kronecker representation.

We describe the major additions to our manuscript, together with changes for readability (iv), in this general response thread, and address specific reviewer's comments in individual answers.

AuthorsOpenReview ↗

i. Language models factorize feedback loops in-context

Reviewers asked about the applicability of our framework to language models, beyond the considered toy models. Inspired by this suggestion, we analyzed whether similar factored representations of feedback loops can arise when evaluating pre-trained language models in-context, following the methodology of the recently published work by Balcells et al. 2026. When evaluating Qwen3.5-9B in context on long token sequences from a feedback loop between transducers of the mess3 and Wing families, results show that the LLM predicts the next tokens about as well as a Bayesian predictor that sees the last two to three tokens (Fig. 5). We observe that the Bayesian beliefs can be extracted linearly from the residual stream at the last layer with held-out R2R^2 of 0.990.99 (mess3) and 0.680.68 (Wing), respectively (App.~N). Finally, we could verify that these factored representations are causally efficacious: on the last layer, patching the belief of one agent along encoders fitted to the per-agent beliefs closes 92.6%92.6\% (mess3) and 70.5%70.5\% (Wing) of the null-to-target gap, i.e. of the KL distance between a write with no effect and the Bayes prediction for the patched beliefs. An encoder of the joint belief of both agents closes less (84.3%84.3\% and 40.0%40.0\%), on all ten sequences, indicating that the additional residual-stream structure it fits is not beneficial to next-token prediction. This is consistent with the LLM reading each agent's belief from its own directions, as the transformers trained from scratch do. The results are summarized in the new Sec. 5, including Fig. 5, with methodological details in the new App. N. Note that as an early investigation into in-context representations of feedback loops, the results remain limited to one example pair of processes and one LLM.

AuthorsOpenReview ↗

ii. Factored encoders steer better than a joint encoder

Since the Kronecker features contain the direct-sum ones, as each marginal is a linear function, per-factor probes alone cannot discriminate between factored and joint representations. Explained variance alone does not favor the factored representation either, as a linear encoder of the Kronecker belief explains more of the residual stream's held-out variance than one of the direct-sum belief (R2=0.88R^2 = 0.88--0.910.91 vs. 0.430.43--0.680.68 in the wide telephone models; App. J).

To differentiate whether the network causally uses structure beyond the marginal distributions, we additionally fit a joint encoder to the residual stream and use it to patch one of the agent's beliefs. We find that the joint encoder fits more of the residual stream's variance, but closes less of the null-to-target gap (95.0%95.0\% for TT, 87.0%87.0\% for UU) than the per-agent encoder (97.6%97.6\% and 98.1%98.1\%), indicating that the additional structure it captures is not beneficial for next token prediction (Sec. 3.3, Fig. 3).

To check that this test can detect a joint code, we applied it to the coarse-graining models (Sec. 4, App. M), whose Bayes belief is a product of per-agent beliefs under full observation but an entangled joint belief once UU's actions are hidden. Patching in a donor sequence's whole belief through an encoder of the agents' marginal beliefs closes 99%99\% of the gap under full observation, but only 68%68\% (5757--85%85\% across five replicates) with UU's actions hidden, where an encoder of the joint belief closes 99%99\% (App. L, Fig. 21). The test thus effectively separates a factored from an entangled representation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.