acceptodds
Under review as a conference paper at ICLR 2027

The Ambiguity Zone: When Randomized Social Preferences Induce Social Inference

Abstract

Randomized uncertain social preferences (RUSP) can produce reciprocity and reputation-like behavior in multi-agent reinforcement learning by randomizing social rewards and introducing structured uncertainty about others’ preferences, but the source of this pressure to understand other agents remains unclear. We propose that there exists a decision-critical region of preference space, the ambiguity zone, in which the best action depends on the partner’s latent type. The location of this zone can be computed before training from the value of social information. Our results support a two-gate account of social inference. The first gate is coverage: across payoff games and training distributions, partner sensitivity increases with the training mass placed on the ambiguity zone (condition-level Spearman , cluster permutation ), and the relation extends to a preregistered fourth game. The second gate is identifiability: a factorial experiment shows zero response outside the ambiguity zone at every information level, while partner-conditioned behavior appears inside the zone only when partner type becomes highly identifiable. A preregistered matched-marginal control reproduces this pattern while holding population cooperation fixed at 0.5. When both gates are open, agents use third-party observations to judge a partner they have never directly interacted with (); resetting the recurrent state removes the effect, localizing this cross-interaction judgment to memory. Finally, a compact RUSP-form bridge that restores randomized reward transformations, asymmetric preference uncertainty, and multi-agent shared-policy learning preserves the same gate ordering at smaller magnitude. Together, these results distinguish the roles of randomization and uncertainty in RUSP: randomization determines which regions of preference space receive training exposure, while identifiable social evidence determines whether that exposure creates pressure to understand other agents.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, social inference, randomized social preferences, opponent modeling, reputation, social dilemmas

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kBLhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper aims to understand the mechanisms behind randomized uncertain social preferences (RUSP), introduced by Baker, 2020. The authors define “the ambiguity zone” as the set of social preference values (α\alpha) for which the value of social information is positive; that is, the values of α\alpha for which knowing the partner agent’s type ZZ increases expected return. They argue that partner-sensitivity in RUSP emerges when 1) there is adequate training coverage of the ambiguity zone and 2) the partner type is identifiable from its action history (which is upper-bounded by the Bayes-optimal identifiability Acc∗(Z∣HL)Acc^*(Z|H_L)). Through experiments in an iterated two-action prisoner’s dilemma setting, they demonstrate that partner sensitivity increases with the amount of training mass McritM_{crit} placed within the ambiguity zone (when partner identifiability is held fixed with deterministic, scripted partners) (Fig. 2), and that partner sensitivity increases with increasing partner identifiability when within the ambiguity zone “critical band”. They also show that behaviors resembling reputation and memory arise from agents observing partner interactions with a third party, and finally demonstrate in the RUSP-bridge that indirect reputation SrepS_{rep} is only nonzero when both the coverage and identifiability conditions are satisfied.

Critical strengths

Rigorous experimentation. The experiments fairly rigorously establish the finding that both gates of coverage and identifiability are required for emergent partner sensitivity. The authors first assess trust sensitivity in response to increasing critical mass for a fixed interaction structure. They then establish that coverage of the game-specific critical zone, rather than certain universal values of α\alpha, is what matters for increased sensitivity by evaluating across games with different payoff matrices (and different ambiguity zone bands). The authors then show that their findings have predictive power by assessing a preregistered fourth game. The authors also demonstrate the interaction between coverage and identifiability (Table 1), and verify via the matched-marginal identifiability control that this isn’t due to changes in the marginal cooperation rate of the partner population.

Critical weaknesses

Organization and information flow. I personally found the paper quite dense and hard to parse; key findings and results were not very cleanly connected to the paper’s main claims and takeaways. Most of the figures in the main text (Figs. 1, 3, 4, 6) are not explicitly referenced in the body of the paper, leaving the reader to infer which findings in the Results section are presented in the figures. The similar section and figure titles helped, but the sections often discussed more results than presented in the figure, making the results unnecessarily hard-to-follow without explicit references to the Figure. (As an example, line 188 says “Figure 2 illustrates two consequences…”; however, Figure 2 appears to only illustrate the latter consequence referred to in this paragraph, the one against a {TFT, All-D} population.) Further, Figure 1’s role in the paper was quite unclear to me; it appears to be connected to experiments described in Appendix F rather than supporting the paper’s main result of the two-gate mechanism behind RUSP. Displaying it in its current prominent position distracts from the primary contributions of the paper. Certain abbreviations and notation were also introduced without explanation; an example is Glow,GmidG_{low}, G_{mid}, and GhighG_{high} in Figure 4. Briefly defining the partner types/strategies (All-D, Grim, etc.) would also be nice for readers who are less familiar with this specific area.

Tie to overall significance / more general problem outside of RUSP. As it is currently written, the paper reads as an in-depth study of certain mechanisms underlying RUSP (Baker, 2020). The motivation for conducting this study of RUSP is less clear to the reader; including some elaboration on this and the significance of RUSP would make the paper more self-contained. Additional discussion about how the paper’s findings might generalize beyond RUSP would also make the paper more interesting to a broader audience. Based on what the paper presents, it’s unclear if the results do generalize to multi-agent RL settings outside of RUSP, which limits the value a broader audience might find in this paper.

Questions for authors

In the section on the RUSP-form bridge (Section 4.4 and Figure 7), it’s somewhat surprising to me that the “original style” condition has zero SrepS_{rep}. For one, its method of broadly sampling α\alpha from U(0,1)U(0,1) resembles the approach of RUSP. Also, both the coverage and identifiability gates are open in this condition, which the paper predicts should induce some sort of partner sensitivity effect. I recognize that SrepS_{rep} is a more demanding test, and would thus be curious if StrustS_{trust} in this same condition is nonzero. If it were, it would strengthen the main claims regarding the two-gate account.

Reviewer_y1tiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks why randomized uncertain social preferences (RUSP) produce partner-conditioned behavior. It defines the value of social information Vsocial(α)V_{\mathrm{social}}(\alpha), the expected gain from knowing a partner's type at reward-mixing weight α\alpha. The ambiguity zone is then defined as the set of α\alpha where this gain is positive, and McritM_{\mathrm{crit}} is the mass a training distribution places on that zone. The paper proposes two "gates" for social inference: coverage (McritM_{\mathrm{crit}}), and identifiability of partner type from history (Bayes-optimal accuracy Acc∗\mathrm{Acc}^{\ast}).

The experiments use PPO agents with GRU-based policies in iterated two-action dilemmas.

Critical strengths

The zone is computed by dynamic programming from the payoffs and partner population, so the predictions it implies (location, sign, and ranking of training distributions by McritM_{\mathrm{crit}}) are well-defined independently of training. The fourth-game results follow the implied ordering (−0.078→+0.150→+0.311-0.078 \to +0.150 \to +0.311), and the matched-marginal control removes a confound (that raising qq lowers overall partner cooperation).

Critical weaknesses
  1. The paper is very hard to parse, which makes its claims hard to check. There are some concrete examples below but this comment is more focused on the storytelling and general flow than the types of specific comments written below.

    • The setup is underspecified. Section 3.1 never defines R,S,T,PR, S, T, P. The reward is written as ri=own+αotherr_i = \text{own} + \alpha \text{other} which is a unusual mathematical formulation.
    • Results are packed into running text as slash-separated number strings (e.g., Section 4.1, "0.000/0.000/0.001/0.043/0.336 for pcrit=0,0.25,0.5,0.75,1p_{\mathrm{crit}} = 0, 0.25, 0.5, 0.75, 1"). Conditions are often named but not described.
    • Terminology drifts. The same region is called the ambiguity zone, critical region, critical band, decision-critical region and trust band (C\mathcal{C}, Cϵ\mathcal{C}_\epsilon, CG\mathcal{C}_G). Off-band, off-critical and off-zone are used interchangeably.
    • Figure 1 illustrates the Appendix F experiments, not the paper's main idea. The introduction, discussion and conclusion repeat the same summary while basic setup goes unexplained.
  2. Related work is thin, and the RUSP claim isn't supported. The bibliography has 18 entries. That is too few for a paper offering a mechanistic account of social inference, and whole relevant literatures are missing. The core finding is that a recurrent agent learns to infer a latent variable when it is reward-relevant and recoverable from history. This is well studied in meta-RL and Bayes-adaptive RL. Also absent:

    • ad hoc teamwork
    • Bayesian theory of mind
    • social learning in MARL

    Against that literature, the two-gate account is close to what one would predict. The novelty then rests on the claim to explain RUSP, and that claim isn't supported.

Questions for authors
  1. Are the two gates actually separated? The paper argues that decision value and identifiability are separate requirements, but the experiments that vary identifiability also vary decision value.

    • At q=0.5q = 0.5 the two partner types behave identically, so knowing the type is both impossible and useless.
  2. Does partner type matter at the bridge's test point? The paper's account says agents should only treat partners differently when the partner's type changes the best action. In the bridge, agents are tested at the identity reward transformation (T=IT = I) against a scripted All-C or All-D partner (Appendix E). If T=IT = I means each agent receives only its own reward, defecting appears to be the best move against both partners, so the account would predict no difference in behavior. Yet the paper reports a small difference and counts it as support. Can you explain why partner type matters at the test point, and how the bridge's McritM_{\mathrm{crit}} values were computed?

Regardless of the answers to Q1–Q2, the paper needs a substantial rewrite for clarity before it can be assessed properly.

Reviewer_g4PNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper identifies social preferences where partner information changes the optimal action. Training coverage of these regions predicts partner-sensitive behavior in controlled games. Partner identifiability is another limiting factor.

Critical strengths

The controlled experiments include useful ablations and transparent reporting of negative results.

Critical weaknesses
  1. The authors selected Reinforcement Learning as the paper's primary area. I find the contribution to this area insufficiently novel. The learner is a standard GRU policy trained with PPO. The ambiguity-zone formulation specializes familiar value-of-information and best-response ideas. The main case for novelty therefore rests on the empirical analysis. I find its additional explanatory and predictive insight insufficiently distinguished from the underlying decision-theoretic intuition.
  2. The broader significance of the findings is not convincingly established. The main analysis centers on how a linear mixture of self and partner rewards shapes behavior in controlled games. The cross-game tests, information controls, and memory ablations strengthen the evidence within these settings. However, the paper provides limited evidence that these findings substantially improve our understanding of reputation learning in richer populations or yield broadly useful training principles. Its significance as a standalone contribution therefore remains insufficiently demonstrated.
  3. The presentation makes the central claims difficult to interpret. Although Section 3 provides formal or operational definitions, the abstract and introduction do not clearly connect "social inference," "partner sensitivity," and "reputation-like behavior" to the concrete task and measured capabilities. Even a reader familiar with reputation may struggle to determine what level of inference is being claimed. Figure 1 does little to explain the main task, the ambiguity zone, or the roles of coverage and identifiability.
Questions for authors

Given that Reinforcement Learning is the selected primary area, could the authors clarify the paper's specific contribution to this field?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.