acceptodds
Under review as a conference paper at ICLR 2027

Intent Inference Produces the Retaliatory Cascades It Prevents

14 views

Abstract

In noisy, repeated games, inferring partner intent can explain an observed defection as an execution error. We study how this ability affects learning and cooperation in Bayesian model-based learners using active inference (AIF) and approximate Bayes-adaptive planning. For symmetric games, we derive how execution noise changes payoff comparisons and the information available about latent intent, and formulate decision boundaries for Bayesian model-based planners. When contrasting these generative model architectures, intent-conditioned learners lose cooperation sharply in Stag Hunt and the iterated Prisoner's Dilemma (IPD). Against an unconditional cooperator in Stag Hunt, raising noise from to reduces mean late intended cooperation from to , compared with to for the observation-conditioned learner, although cooperation remains the better one-round response to the known partner. AIF learning traces show that runs with different eventual outcomes can have similar beliefs about current partner intent but different predictions about whether mutual cooperation will continue. We hypothesise that defection helps preserve unfavourable predictions by reducing opportunities to test continued cooperation. Observation-conditioned clusters are well described by learned responses to executed outcomes; some intent-conditioned AIF runs instead alternate rapidly between C and D. Bayes-adaptive mutual cooperation is lower in IPD self-play than against Tit-for-Tat, and mainly gives way to asymmetric play in Chicken. Both intent-conditioned planners retain high intended agreement in Pure Coordination self-play. These findings motivate reporting per-run behaviour and intended and executed joint actions alongside mean cooperation. They also motivate research into how inferred intent could guide exploration toward poorly learned transitions and stabilise learning under execution noise.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Game Theory, Intention Inference, Bayesian Methods

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BHD7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines cooperation in repeated games where execution noise can flip an agent’s chosen action. It compares active-inference learners that infer latent intentions with learners that model executed outcomes directly, alongside approximate Bayes-adaptive and Q-learning baselines. The analysis covers Stag Hunt, Prisoner’s Dilemma, Chicken, and Pure Coordination.

Intent-conditioned AIF learners sometimes lose cooperation much more sharply than observation-conditioned learners, even against an unconditional cooperator. The authors find that learners can recognize a partner’s cooperative intent while remaining pessimistic about whether cooperation will continue. They suggest that this pessimism discourages cooperation, limiting the experience needed to revise it. Trajectory analyses and interventions test parts of this explanation.

Critical strengths

The unconditional-cooperator experiment provides a useful control. The learner loses cooperation even though its partner never deliberately defects and cooperation remains the better one-round response below the known noise threshold. The shared AIF settings, substantial seed counts, and fresh-seed replication support the reported difference between representations.

The trajectory analyses explain behavior that average cooperation rates would miss. Extracted response rules account for the observation-conditioned clusters, while some intent-conditioned runs alternate rapidly between actions. Reporting intended and executed actions separately is also helpful, particularly in Pure Coordination. I appreciate that the authors report mixed intervention results, including the limited recovery in Appendix F.8.

Critical weaknesses
  1. The cause of cooperation loss remains unclear. Lower cooperation itself reduces evidence about cooperative transitions, so the trajectory analyses do not establish the direction of causality. The information-value ablation addresses how cooperation starts, while the recovery intervention in Appendix F.8 improves predictions without reliably restoring cooperation at the primary noise level. The evidence therefore does not sufficiently distinguish a failure of intent modelling from effects of approximate learning or the planning objective.

  2. The contribution and broader significance need clearer justification. Prior AIF work already examines cooperation oscillations and exploration-dependent learning. Although the execution-noise comparison is interesting, the paper does not clearly establish what new insight transfers beyond the tested planners. A concrete motivating application would help explain which practical difficulty the model captures and how the findings should influence the design of cooperative agents.

Questions for authors
  1. What keeps the full planner from returning to cooperation in Appendix F.8? Could a controlled change to the cooperative-transition prediction separate its effect from changes in information incentives?

  2. Which results show cooperation breaking down after it has become established? How is a retaliatory cascade distinguished from persistent defection or self-generated alternation between C and D?

  3. What does the analysis establish beyond prior AIF accounts of oscillation and exploration-dependent cooperation? In what concrete coordination setting would the authors expect the same failure?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BHD7

We thank the reviewer for the thorough review.

To address the weaknesses:

  • On the cause of cooperation loss. We agree that the trajectory analyses are retrospective and cannot determine the direction of causality on their own because the explanation itself is cyclic in the sense that a pessimistic prediction discourages cooperation and less cooperation means less evidence for cooperation. The experiments were done to separate these points. Removing the information bonus changes what the learner does and therefore what it gathers information about. Explicitly supplying cooperative evidence changes the prediction itself. What is left is the closing of that loop from an improved prediction back into cooperation. We will make this clear in the paper.
  • On distinguishing intent modelling from approximate learning and the planning objective. The planning objective is the same across the two active-inference learners. The Bayes-adaptive learner, which plans differently and has no explicit information bonus, also loses cooperation sharply (Figures 7 and 8). The inference of intent itself does not fail, learners that later defect are about as confident in the cooperative intent as those that later cooperate, but what differs is instead how they predict the future of that interaction. We have not separated it from a more exact update, and we will limit our claims to learners of this kind within the revision.
  • On the contribution and broader significance. The insight we expect to transfer is the idea that inferring intent separates two types of beliefs, one about what the partner intends now and one about what follows. One can be correct while the other is wrong. The loss of cooperation is not due to one planner since both planners we test show it. For the design of cooperative agents it means that intent inference alone is not enough, and that there needs to be some way to incentivise the agent to test cooperation on-policy, likely through some hierarchical mechanism that we leave to future work.

To address the questions.

  • On what keeps the planner from returning to cooperation. Our exploratory analysis in the appendix points to two main things. The forced evidence does improve the cooperative prediction, but the payoff advantage of cooperating stays negative. The added evidence also reduces the information bonus that supports cooperative exploration. The controlled change to the prediction that the reviewer suggests would separate these two effects. We have not run it but will note it in the paper as a natural next experiment.

  • On breakdown after cooperation has been established. Our results do not show this. In our samples, no run that was already cooperating early on later met our low-cooperation criterion (Appendix F). We will revise the wording and title that may obscure that.

  • On telling the behaviours apart. We look at the action sequences within the trajectories. Persistent defection is the low cluster. Self-generated alternation is the intermediate cluster, where the learner switches their action on most rounds while remaining confident that its partner intends to cooperate. Against an unconditional cooperator neither is ‘retaliation’ in the usual sense. The alternation mirrors a retaliatory cycle in traditional game theory, but the partner never actually retaliates, so it is entirely self-contained. For Tit-for-Tat and self-play we report cooperation rates and have not analysed response sequences, and we will state this in the revision.

  • On prior AIF accounts. The AIF work we cite studies conditions for cooperation in the Prisoner's Dilemma without execution noise [4], and models of opponents' internal states in iterated general-sum games under changing payoff structures [5]. What our analysis adds is the comparison under noise where the same planner has the noise channel either in its observation model or folded into its transition prior, against varying types of partners, and isolating the focal agent’s contribution to that interaction. If the reviewer has a specific account of oscillation in mind, we would be glad to position our work against it.

  • On a concrete setting. We would expect the same failure in a spatial Stag Hunt or level-based foraging under action noise, where agents must coordinate on a risky but rewarding option under a form of execution noise that corrupts movement. More generally it should arise where there are coordination outcomes of different value, a latent variable learned through a noisy channel, near-greedy action selection, and rewards on observed outcomes. We treat these as future work.

AuthorsOpenReview ↗

Continuation of comment

[4] Demekas, Daphne, Conor Heins, and Brennan Klein. "An analytical model of active inference in the iterated prisoner’s dilemma." International Workshop on Active Inference. Cham: Springer Nature Switzerland, 2023.

[5] Jaime Ruiz-Serra, Patrick Sweeney, and Michael S. Harré. 2025. Factorised Active Inference for Strategic Multi-Agent Interactions. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS '25). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1793–1802.

If our responses address your concerns, we would be grateful if you would consider raising your score. We would be happy to continue this discussion on any remaining points.

Reviewer_UzqwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how inferring a partner's intent affects learning in symmetric 2-player games. It first establishes, in formal terms, how noise distorts the information about partner intent; then, it models, with active inference and approximate Bayes-adaptive planning, how cooperation evolves in repeated games and how it affects cooperation. The paper reports experiments on the stag hunt game and shows that cooperation in intent-conditioned settings drops heavily when the amount of noise reaches a threshold.

Critical strengths

The paper is well written and, despite not my area of expertise, easy to follow with the contributions clearly stated. The topic and setting are those classics of game theory, with players that learn and adapt to each other's responses. I appreciate the introduction (more than the abstract, see weaknesses), as it motivates well why the setting is relevant and of interest for the game theory community.

Results are interesting (but please also read weaknesses): in Stag Hunt, noise has only a marginal effect on collaboration with observation-conditioned active inference, while it makes collaboration collapse with intent-conditioning. The explanation the authors give is interesting, i.e., that intent-conditioning reduces the opportunities to test the cooperative transition.

Critical weaknesses

(minor) I think that the abstract would benefit from numerical results being moved to the relative section and replaced by a summary of the observations. On the one hand, I appreciate that the authors do not over-state and generalise their punctual observations; on the other, it makes me think that the results may not hold in other settings and may be an artefact of theoretical modeling.

(major) Observing the results (and Figure 1), it is not clear whether the two update methods induce a change in the game that actually reflects the intent of the other player, as they sharply diverge. My main concern is that the comparisons do not properly isolate the effect of intention inference. The two approaches differ not only in their representations but also in their inference approximations, learning dynamics and information-seeking objectives. So the observed failures could simply be artifacts of the particular implementations rather than a fundamental problem with intention inference. The explanation given by the authors involving pessimistic beliefs and insufficient cooperative exploration is plausible, but the authors' own intervention experiments provide limited support for it. I think there is a potentially worthwhile observation here, but the novelty is modest, the central claims are somewhat overstated and stronger ablations and more carefully controlled experiments are needed to make the conclusions convincing.

(major) The paper has an interesting empirical observation, but the novelty and overall contribution are limited. The main ideas are already well established in the literature. This includes inferring intentions under execution noise, cooperation breaking down due to errors and self-reinforcing learning dynamics caused by insufficient exploration. The mathematical parts are also fairly straightforward. What is potentially interesting is the finding that an agent explicitly modelling intentions can perform significantly worse than one learning directly from observed actions, even against an unconditional cooperator.

(major) While stag hunt is a classic and impactful game, the experiments in the paper reduces to only that setting, severely limiting the impact of the work. There are plenty of games the authors may test and, since the learning phase does not involve complex architectures, that would be the first major area of improvement of the paper.

I conclude by saying that this is not my direct area of expertise, so I'd be happy to read other reviewers' comments and eventually change my mind about the paper.

Questions for authors

Q1. In [1], agents can maintain incorrect beliefs about situations they never encounter because their chosen actions do not generate evidence contradicting those beliefs. This is almost exactly the conceptual mechanism proposed in the paper. In [2], agents learn from observed play, but incorrect beliefs about unvisited parts of the game can persist indefinitely. Those are two examples of (quite old) papers that overlap with the authors' explanation, but the authors' statement that latent-intention inference amplifies it relative to observation-conditioned learning is not something that was fully established yet Can the authors establish their contribution in terms of these existing results in game theory and MAS?

Q2. Why did the authors not extend the analysis to other games, since the cost to run the experiments would not be that heavy?

[1] Fudenberg & Levine (1993), Self-Confirming Equilibrium [2] ⁠Fudenberg & Kreps (1995), Learning in Extensive-Form Games I: Self-Confirming Equilibria.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Uzqw

We thank the reviewer for the kind and careful review.

To address the weaknesses:

  • On the numerical results in the abstract. We agree that the abstract would benefit from a more meaningful summary of the observations and we will do that in the revision.
  • On isolating the impact of intent inference. The agents do differ in how evidence updates the beliefs and model, but this difference follows from the representation. Once intent is treated as latent, observations must be assigned to transitions through the inferred latent, which we do with soft counts. This is a consequence of latent intent modelling, and is the mechanism that we are studying. Other settings, such as the horizon, planning objective and learning rate, are shared. The learners do recognise the partner's intent. Against the unconditional cooperator, runs that later defect and runs that later cooperate are about equally confident that the partner intends to cooperate (Appendix F).
  • On the support from the interventions. We agree that our explanation is a hypothesis and that the interventions support some parts of it more strongly than others. They establish that what the learner does determines what it learns. They also establish that cooperative experience improves the learner’s prediction that cooperation will continue. They do not establish that the improved prediction is enough to bring the learner back to cooperation, which seems to depend on the level of noise. We will make this distinction clear in the paper and update the abstract and title in line with that.
  • On novelty. We agree that the individual pieces that make up the study are established, and we cite the work on execution noise and intent inference in Section 2. As you have noted, the new observation is that intention modelling does worse than observation modelling under meaningful noise which serves as the core contribution of the paper. The derivations shown in section 3 aim to show that this observation is not caused solely by lost information due to noise, or myopic decisions.
  • On the experiments being limited to Stag Hunt. The paper also covers the Iterated Prisoner’s Dilemma, Chicken and Pure Coordination, each against Tit-For-Tat and in self-play (Section 6) and Figure 1 is the Prisoner’s Dilemma. The detailed analysis in Section 5 is on the stag hunt only and the cross-game figures are in the appendices, which may hide the scope of the contribution. We will move a more detailed cross game summary into the main text. The sharp intent-conditioned loss extends to Prisoner’s Dilemma, while in Chicken lower cooperation mainly gives way to asymmetric play, and in Pure Coordination intended agreement stays high in self-play.

To address the questions.

  • On the relation to self-confirming equilibrium. Thank you for the question. This is a line of work with a shared intuition and was a core part of the paper’s development. We had attempted a formal mapping but found that it did not really work without extensive development of the theory. The main issue involved is the scaling from the clean setting in which [1,2] operate to the noisy setting we operate in but the literature surrounding the bridge from clean to noisy settings is quite sparse. The main intuition for why the approach was discarded is that the true evidence is never seen by the agent, and it has to assign evidence based on its experience, whereas in [1] the requirement that beliefs are correct on the path of play “implicitly supposes that players observe the terminal node of the game at the end of each play”, which does not hold here. This already puts a very significant limitation on how much of the framework can be translated meaningfully. We did do a deeper look into things like Berk-Nash Equilibria [3], which does some of the work involved in scaling the idea to games where players can only observe outcomes and identifying conditions for reduction into SCE. The main limitation here is that their definition of a misspecified model isn’t what we have, but it does mean that the connection is still an open question. We hope to revisit these ideas soon.
  • On extending the analysis to other games. The cross-game results are in Section 6 and Appendix G, as described in the response to the weaknesses. The trajectory analysis of Section 5 has only been done for Stag Hunt, and we will state this as a limitation.

[3] Esponda, I., & Pouzo, D. (2016). Berk–Nash equilibrium: A framework for modeling agents with misspecified models. Econometrica, 84(3), 1093-1130.

If our responses address your concerns, we would be grateful if you would consider raising your score. We would be happy to continue this discussion on any remaining points.

I appreciate the author's response. Somehow, the authors confirmed the limitations of their work (every work has some limitations and cannot cover everything, though), including a not-very-extensive empirical evaluation and some issues with the novelty. At the same time, this work has some merits: above all, the problem the paper addresses is important and quite foundational for other fields.

While I do not believe there is ground to increase my score to 3 (the system this year is limiting us to choose between a very narrow range of scores), I ask the AC and SAC to consider my evaluation more positive than my initial review, something closer to borderline accept, and decide based on this and the other reviews and rebuttals.

Reviewer_zAp5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether inferring a partner's latent intent—treating an observed defection as potentially a noise-corrupted execution of a cooperative intention—actually helps cooperation in noisy repeated games. The authors study Bayesian model-based learners using active inference (AIF) and an approximate Bayes-adaptive POMDP baseline, in symmetric 2×2 games with a binary symmetric execution channel of known crossover rate ε.

Critical strengths
  1. The surprising result is real, not a fluke. They reran the whole thing on 256 fresh random seeds per noise level and got the same contrast. It's not one lucky run.

  2. They found the actual cause, not just the symptom. The collapsing and cooperating agents believe almost the same thing about their partner right now (91% vs 91%). What separates them is their prediction about the future — and the collapsing agent has less evidence for that prediction because it keeps defecting. That's a clean, specific mechanism.

  3. The "remove curiosity" test is decisive. Take out the information-seeking term and every single run defects forever. That pins the cause on one specific piece of the algorithm rather than hand-waving about "intent inference" in general.

  4. They explain the messy middle instead of ignoring it. The half-cooperating runs aren't a stable compromise — they're flipping back and forth almost every round. The simpler agent's middle runs are explained by fixed habits the authors extract and verify predict 99%+ of later moves. That turns a confusing cluster into a concrete behavioral story.

Critical weaknesses
  1. The comparison changes two things at once. The "intent" agent differs from the "no-intent" agent not just in what it tracks, but in how it updates its beliefs. The authors are upfront that this is an approximation, but a reader can't tell how much of the collapse is from tracking intent versus from the specific shortcut used to do it.

  2. The big collapse might depend on settings they didn't vary. The headline experiment uses a short planning horizon and one update schedule; the cross-game experiments use different ones. They don't show whether the 93% → 20% drop survives if you change those knobs. Since the sharpness of the collapse is the headline, that's a real gap.

Questions for authors

Would a different shortcut change the result? The intent agent uses one particular approximation. If you swapped in a more accurate method for tracking intent, would the collapse shrink, stay the same, or get worse?

How fragile is the headline number? The 93% → 20% drop happens over a small noise change. Does it survive if you change the planning horizon, how often the agent updates, or how strongly it starts out believing one thing?

Can a collapsed agent be rescued? The rescue test came back inconclusive. What's the strongest evidence that the vicious cycle keeps an agent down, rather than just getting it into trouble once at the start?

1 comment
AuthorsOpenReview ↗

Response to Reviewer zAp5

We thank the reviewer for the precise review. To address the weaknesses.

  • On isolating the impact of intent inference. The agents do differ in how evidence updates the beliefs and model, but this difference follows from the representation. Once intent is treated as latent, observations must be assigned to transitions through the inferred latent, which we do with soft counts. This is a consequence of latent intent modelling, and is fundamentally the mechanism that we are studying. Other settings, such as the horizon, planning objective and learning rate, are shared.
  • On the dependence on settings. The headline experiment and the cross-game experiments were tuned separately, each taking the best performing configuration on a smaller portion of seeds by their mean mutual cooperation rates, with those configurations being carried over into the main study as representative settings. Table 1 reports only the cross-game tuning grid, and we will add the separate grid used for the unconditional cooperator benchmark. Under other settings cooperation did not really emerge as strongly and we will state this in the paper. The sharp falloff occurs in both the Prisoner’s Dilemma and the Stag Hunt (Section 6) and under the two configurations used in the paper. We can include a dedicated appendix showing the pattern over varying parameters.

To address the questions.

  • On whether a different approximation would change the result. We have not tested a more accurate update, so we cannot say which way it would go. If the reviewer has anything in mind we would be keen to discuss this further.
  • On the fragility of the headline numbers. Beyond the experiments above, we have run an ablation with tempered action selection (not included in the submission) and we found that softening the policy does make the collapse less sharp but effectively shifts it to a lower noise level with reduced cooperation across noise levels. This is due to action tempering sharing the behavioural pattern with environmental noise. We have not varied the prior strength.
  • On rescuing a collapsed agent. The strongest evidence that the cycle keeps a learner down comes from what happens without doing any intervention. The learners that are defecting by round 1,000 are still defecting at the end (Appendix F). Across the run, the prediction of continued cooperation levels off early for the group that ends up defecting, while it keeps improving for the group that ends up cooperating (Figure 6b). And in runs that do move between the two behaviours, periods of defection last far longer than periods of cooperation (Figure 4).
  • On rescue itself. Forced cooperative evidence improves the prediction, but its impact on rescuing the agent is only partial. At a slightly lower noise level, the agents that learn from the forced evidence show increased cooperation, but at higher noise levels cooperation does not recover. Our analysis suggests this is because the payoff advantage of cooperating stays negative. To deal with this we tried some symmetric forgetting approaches (also not in the submission), but even in the best cases they made cooperation worse.

If our responses address your concerns, we would be grateful if you would consider raising your score. We would be happy to continue this discussion on any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.