acceptodds
Under review as a conference paper at ICLR 2027

LatentBot: Replacing Source Code with Activations in Program Equilibrium

Abstract

LLM agents increasingly interact in settings where cooperation depends not only on observed actions but on anticipating how other agents will behave. Classical program equilibrium solves this by letting agents read one another's source code, but an LLM's weights do not reveal the strategy it will follow in a given interaction. We introduce LATENTBOT, which replaces source-code inspection with activation inspection: it extracts an opponent's pre-action hidden state and uses lightweight probes to estimate both the opponent's immediate action and its counterfactual reaction to cooperation or defection. These forecasts are combined with interaction history and short-horizon utility estimates and passed to the acting LLM, which retains the final decision rather than following a fixed strategy. Across six games from CoopEval, LATENTBOT converges to sustained mutual cooperation in self-play while adapting to competitive, suspicious, and reciprocal opponents, and plays competitively in zero-sum Matching Pennies. Probes recover an opponent's counterfactual next action with 93.6% accuracy before it is emitted. The signal transfers across model families: a 128-dimensional linear alignment from Mistral into Qwen's latent space preserves a frozen Qwen probe's predictions without any Mistral game labels. A behaviorally matched surrogate extends the method to an output-only commercial model. Internal activations therefore offer a coordination channel between independently trained agents beyond what their actions and messages alone provide.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, multi-agent cooperation, program equilibrium, opponent modeling, activation probing, representation alignment, social dilemmas, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_p7BsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an agent/methodology called LatentBot, which uses an opponent LLM's internal activations to learn strategic action decisions. The methodology is as follows: The history of rounds and the current round's prompt are fed to the opponent LLM. Then, before the opponent LLM outputs, LatentBot extracts the activations of the opponent LLM from a certain intermediate layer. It trains a lightweight classifier to predict the opponent's action from the latent activations. The experiments show that LatentBot has highly desirable performance in predicting the opponent's actions.

Critical strengths
  1. Using activation probes as an input to another agent’s decisions moves probing beyond offline analysis and raises useful questions about communication between neural agents.

  2. Methodologies such as training a regression model between different LLM activations to transfer activations, probing the activations using lightweight models and their success can be useful and transferable to the interpretability of LLMs.

Critical weaknesses
  1. The activation-access assumption makes the prediction task substantially easier and limits applicability. LatentBot reads the opponent’s layer-14 activation at the final prompt position, directly accessing an intermediate computation that produces the opponent’s action. Predicting that action is therefore substantially easier than inferring an unknown strategy from observable interactions. Moreover, the method requires access to the opponent’s model and execution context, restricting its applicability. Also, since the activations need to be captured basically right before the output. I don't see the practicality of extracting the activations right before the output instead of simulating the LLM directly.

  2. The extraction-layer choice is insufficiently justified. Although the transfer experiment mentions validation-based layer selection, the native and surrogate settings lack a documented selection procedure and a layer-wise sensitivity analysis.

  3. The counterfactual reaction prediction simulates the current round and the next round. It has one step of lookahead. In a more complex game setting, a game tree might need to be gone through to search the best direction given the paper's methodology, which can be time consuming. The authors can consider how the agent should explore/learn the best outcome in a more complex setup.

  4. It seems the black-box experiment assumes substantial knowledge of the opponent and does not establish recovery of its internal strategy.

Questions for authors

Q1: Can the authors justify the d

Reviewer_Tr3qOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper lets an AI game player look into the internals of its opponents just before the opponent makes a move in order to predict what it will do. The aim here ius to cooperate with partners who cooperate back while protecting itself from cheaters. In games against itself's copies, it does cooperate well.

interestingly, the player never cooperated with any opponent other than itself (it betrayed everyone else in every round, including opponents that cooperate when cooperated with). The opponents are never described, and some behave nothing like their names: “CooperateBot,” for example, betrays in most rounds.

The method already runs the opponent’s model, so the obvious comparison is to simply ask that model what it would do, and this is missing. The “black-box” version, meant to work without inside access, was actually told the opponent’s strategy in plain text.

Critical strengths

The premise of the paper is very interesting. I will paraphrase it as can looking inside another AI’s workings take the place of reading its code, as a way to build trust between agents?

In games against copies of itself the player cooperates reliably, while the same player without the inside look falls apart partway through. The method is clearly described, and the authors are open about several of its failures.

Critical weaknesses

The paper’s own scores contradict its main claim. Working through the results for the classic cooperation game shows the player never cooperated with any opponent other than itself.in fact, It betrayed everyone else in every round, including opponents that cooperate when cooperated with. Against an opponent that always cooperates in another game, it chose the safe option almost every round and gave up the much larger shared reward.

Questions for authors
  1. How is each opponent built? For the scripted ones, whose internal workings is the player looking at?
  2. Do your scores mean the player never cooperated with any opponent except itself? If so, how is that different from just cooperating with copies of yourself? 3 . What does looking inside the opponent add over simply running the opponent’s model and seeing what it does?

Further, maybe you could test whether an opponent can look cooperative on the inside and then betray anyway.

Reviewer_GqS9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors develop a LLM based player of 2x2 matrix form games that reads the latent states of its counterparty and tries to predict the next and hypothetical future actions. This agent is evaluation against some baselines in repeated games.

Critical strengths

There is something interesting here. Neural models are transparent in an interesting way that could be relevant for strategic interactions and this is the first attempt I’ve seen to characterize strategic interaction under this kind of transparency.

Critical weaknesses

Paper is poorly written and hard to follow. There is no motivation for this work. Why is it important? The idea of “Activation-Based Program Equilibrium” as studied is invalid. How is this related to programs or equilibrium in activation space? Its just a modified game where players are transparent to some extent. Halpern, J. Y., & Pass, R. (2018). Game theory with translucent players: JY Halpern, R. Pass. International Journal of Game Theory, 47(3), 949-976. Not clear how it would generalize to sequential games e.g., CoinGame

Questions for authors

Figure 3 — looks like basebot fails to cooperate before there is any defection? The line goes down before the first open circle.

Authors talk about Activation-Based Program Equilibrium but there is no notion of strategy at the level of activations which is what the analogy would imply. To what extent is equilibrium assessed anywhere in this paper?

It’s hard to tell if the predictive success is just due to the models being really easy to predict from history. How much do the activations even help predict the behaviors? One should be highly skeptical of results like this: “On the held-out Prisoner’s Dilemma data, the probe predicts opponent’s current action with 100% accuracy on all samples”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.