acceptodds
Under review as a conference paper at ICLR 2027

NormGym: Evaluating Normative Competence through Social Interaction

Abstract

Normative competence, the ability to infer socially enforced expectations from interaction and adjust behavior accordingly, is essential for agents operating in unfamiliar human communities. Yet existing evaluations largely test whether agents can recognize or apply norms they may already know, rather than whether they can acquire previously unknown norms from social interaction. We introduce NormGym, an interactive benchmark for normative competence in which a newcomer must infer a hidden, arbitrary norm while pursuing its own task in a multi-agent community. Success requires distinguishing what is possible from what is permitted, inferring the norm from observed behavior and sanctions, and applying it in new situations. NormGym contains diverse norms and distinct environments, allowing systematic evaluation of current agents, including large language model (LLM) agents. We find that normative competence remains challenging: the strongest evaluated LLM achieves a joint task-and-compliance success rate of 55.3%. Our analysis reveals that models can sometimes revise their behavior following sanctions for their own actions, but struggle to use other agents’ behavior and the sanctions they receive to infer the underlying norm. We further find that fine-tuning models may be insufficient to improve normative competence. These results suggest that current models lack a robust, transferable capability for acquiring normative knowledge through social interaction.

Primary area:
datasets and benchmarks
Keywords:
normative competence, multi agents, social norms, social learning, LLM, embodied agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jVQkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces NormGym, a multi-agent interactive benchmark for normative competence evaluation. The main purpose is to evaluate competence under norms that agents are unlikely to have seen during pretraining to test adaptability to new environments with custom social norms. For that purpose, they design 50 “silly rules”, described as arbitrary norms.

The suite is composed of 10 layouts (e.g. a cafe). Each episode is assigned a layout and a hidden norm, and all agents have two consecutive goals. The agents can take actions that violate the norm. However, such actions trigger an environment-issued, globally visible sanction.

The evaluation consists of two main steps: interactive learning, and post-interaction evaluation. In the former, an agent interacts with a fixed layout for 200 steps, simultaneously inferring the norm and taking actions to achieve its goals. For the post-interaction evaluation, the agent is given a frozen belief, and its performance is evaluated both in the “training” layout, and an unseen one.

The authors evaluate 3 closed source models and 4 open-source models, concluding that even the strongest models at the task struggle to infer the social norms from interaction. The agents are said to mainly learn through their own sanctions, rather than from observing other agents’ behaviours. Lastly, they attempt training Qwen3-VL-4B-Instruct on normative competence subtasks and show little impact on NormGym performance.

Critical strengths
  • The authors collect human data on their benchmark to verify that the tasks are solvable and to give an indicator of what good performance may look like. Additionally, to separate norm inference from execution consistent with the norm, they test the performance of GPT-5.6 Sol when given the true norm of the environment.
  • The manuscript includes a process-level analysis of belief inference. Rather than simply reporting performance on the benchmark, the authors analyse the belief acquisition trajectory throughout the interaction, and investigate how different sources of evidence (own violation, observed violation) impact belief change.
Critical weaknesses
  • Section 3.3 states that agents receive a textual state description including, among other things, all agents’ sanction signals. However, no such information seems to be included in the Appendix H prompts. My current understanding is that agents receive information about other agents’ sanctions purely from a red cross on top of the sanctioned agent in the rendered frames, and that these are not recorded in memory. By contrast, the agent’s own sanctions are included in the text, flagged in the 10 most recent steps in memory, and in a buffer containing up to 5 of the sanctioned earlier steps (Appendix C.2). My concern with this is that one of the main conclusions of RQ2 is that agents mainly learn norms via their own sanctions. However, if the setup is as I described above, there is a clear asymmetry in the way that both sources of evidence are presented to the agent, with its own sanctions being much more salient.
  • Each norm has a single interactive learning phase, whose final belief is frozen for the evaluation phase. The 6 test episodes therefore measure the variability in belief execution rather than belief acquisition, even though studying this acquisition is the main purpose of the benchmark. Additionally, Table 1 reports no confidence intervals, making it difficult to interpret differences across models and types of norms.
Questions for authors
  • Could the authors state clearly what information the agent receives via text (including per-step prompt and memory) vs through the rendered image? And, if the setup is as described in my first weaknesses bullet point, could the authors run an ablation in which other agents’ sanctions are presented in the same way as the agents’ own sanctions, and report the resulting change in belief correctness per source of evidence (same as Figure 4c)?
  • Could the authors report confidence intervals for the main results in Table 1? And, if there is one single interaction phase per norm (i.e. one belief per evaluation), could the authors run multiple interaction phases to estimate the variability in belief acquisition?
6 comments
AuthorsOpenReview ↗

W1, Q1 — Missing information about other agents' violations in the prompt

Thank you for pointing this out. To address this, in the prompt we added an explicit note about other agents' violations everywhere the agent is told about its own (though the other agents’ violations have already appeared in the rendered image):

  1. The current-step prompt now says "In the previous step, [agent name] violated the social norm, so [name] now has a norm-violation marker and is being penalised in this step," next to the existing note on the agent's own violation.
  2. Each step in the 10-step recent memory now lists which other agents violated the norm.
  3. The persistent record of earlier violations now also says which other agent violated the norm, not only the agent's own.

Nothing else changed, and we re-ran GPT-5.6 Sol on all 50 norms.

The same pattern remains after these changes. The table shows the average change in belief correctness immediately after each type of event (same measure as in the paper Figure 4 panel c), reported as mean ± SE. Explicitly including other agents' violations in the prompt and memory increased belief improvement after observed violations from 0.023 to 0.057. However, the model still updated its beliefs more strongly after its own violations (0.162), while belief changes following no violation remained minimal (0.004). This shows that the difficulty of learning from others is not solely due to how violation information was presented in the original prompt. We have added this ablation experiment and its results in Section 4.4.2 (lines 421 to 426) and Appendix D.3, with the modified prompt and memory format provided in Appendix H.1.

Table 1. Effect of explicitly recording other agents’ violations in the memory on belief revision

ConditionOwn violationObserved violationNo violation
Original (data reported in the paper, without other-agent-violation note)0.23 ± 0.100.023 ± 0.0130.005 ± 0.003
New (with other-agent-violation note)0.162 ± 0.0610.057 ± 0.0160.004 ± 0.002

We also checked whether these prompt changes affected the model's overall performance. Joint success remained similar, changing from 44.0% to 48.7% on seen layouts and from 40.7% to 44.0% on unseen layouts. Paired Wilcoxon signed-rank tests across all 50 norms showed no statistically significant difference in joint success between the original run and the new run, on either seen layouts (p = 0.34) or unseen layouts (p = 0.43). Even with the additional information about other agents' violations in the prompt, the model still performs poorly. This gives us additional confidence that our overall findings are not driven solely by how other agents' violations were presented in the original prompt.

AuthorsOpenReview ↗

W2, Q2 — Acquisition variability

(1) Thank you for raising this point. We would like to clarify that the learning phase is not a single episode, but it is a continuous process over 200 steps where the agents can update their belief about the norm at each step. At every step, the agent gets new evidence from their action, and there are new background agents coming in with different identities and goals. This gives the agent repeated opportunities to observe different situations and refine its understanding. We set a fixed 200-step budget to keep the learning conditions consistent across models and norms. We have clarified the evaluation protocol in Appendix C.1 of the revised manuscript (lines 1058-1062).

We also understand this is a valid concern, so we ran an additional experiment to further check the consistency of norm acquisition. We re-ran the 200-step interactive learning phase of GPT 5.6 Sol on 20 norms, randomly selecting 5 from each norm family. We used the same environment and seed as in our new ablation experiment above with the other-agent violation information. We then used the LLM judge to score the final beliefs from both runs on the same 1-7 scale. The average score was 3.65 in both runs, with no significant difference between them (paired Wilcoxon signed-rank test, p = 0.89). This shows that norm acquisition is reasonably consistent across repeated learning runs. We have also added this repeated-learning experiment and its results in Appendix D.5 of the revised manuscript.

(2) We would also like to clarify that the purpose of having six test episodes here is not just to evaluate whether the agent can execute its final belief, but to test whether what it learned during interaction can generalize to new situations. In other words, we want to see if it really learns the norm, not just memorizing patterns and behaviors from the learning phase. We have 3 test episodes that are in the same layout seen by the agent in the learning phase, and 3 test episodes that are in a different layout. For each test episode, all agents in the environment have different initial positions and goals. By having independent test episodes with different initial setups, we can know whether the agent has learned the underlying norm. We design it this way to eliminate the possibility that agents just memorize specific behaviors. We have clarified the evaluation protocol in Appendix C.1 of the revised manuscript (lines 1083-1088).

In our experiments, we also did the Know-Norm baseline, where the model is directly given the true norm, to help separate failures in learning the norm from failures in executing it, which indeed shows that the model can achieve near-perfect joint success when given norms, but not when not given them.

AuthorsOpenReview ↗

W2, Q2 — confidence intervals

We report 95% percentile-bootstrap CIs over norms in Appendix D.4. Human results cover 25 randomly selected norms and report the best observed performance per norm, so no CI is reported. These intervals help show the uncertainty in the performance estimates, so we should be cautious about drawing strong conclusions from small differences between models or norm families. At the same time, our main finding remains clear that learning and following unfamiliar community norms remains challenging.

Table 2. Performance on NormGym (%). Each cell shows performance in seen / unseen layout, as percentages with 95% CIs in brackets. The columns Access, Procedure, Coordination, and Allocation report joint success within each norm family.

BaselineAccessProcedureCoord.Alloc.Joint Success, allTask Success, all
Claude Opus 569.4 [50.0, 88.9] / 69.4 [44.4, 91.7]48.5 [21.2, 75.8] / 54.5 [27.3, 81.8]66.7 [47.9, 83.3] / 54.2 [37.5, 70.8]30.3 [21.2, 39.4] / 24.2 [9.1, 39.4]55.3 [44.7, 66.0] / 51.3 [39.3, 62.7]95.7 [91.7, 98.7] / 96.7 [92.7, 99.3]
GPT-5.6 Sol69.4 [52.8, 86.1] / 44.4 [22.2, 69.4]45.5 [21.2, 69.8] / 57.6 [30.3, 84.8]20.8 [8.3, 33.3] / 29.2 [18.8, 41.7]48.5 [27.3, 72.7] / 36.4 [12.1, 60.6]44.0 [33.3, 54.7] / 40.7 [30.0, 52.0]86.3 [78.3, 93.3] / 83.0 [74.0, 91.3]
Gemini 3.1 Pro36.1 [11.1, 63.9] / 36.1 [13.9, 61.1]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]37.5 [16.7, 58.3] / 14.6 [6.2, 25.0]15.2 [0.0, 36.4] / 15.2 [0.0, 33.3]24.0 [14.0, 35.3] / 16.7 [8.7, 25.3]90.3 [82.7, 96.7] / 87.7 [79.3, 94.7]
GLM-4.6V-Flash0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]11.7 [8.3, 15.7] / 15.7 [11.0, 20.7]
Qwen3-VL-8B5.6 [0.0, 16.7] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 3.0 [0.0, 9.1]1.3 [0.0, 4.0] / 0.6 [0.0, 2.0]38.3 [31.7, 45.3] / 34.0 [26.7, 41.3]
Llama-3.2-11B0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]3.0 [1.0, 5.0] / 3.0 [1.3, 5.0]
Ministral-3-8B8.3 [0.0, 16.7] / 8.3 [0.0, 16.7]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 2.1 [0.0, 6.2]6.1 [0.0, 15.2] / 6.1 [0.0, 15.2]3.3 [0.7, 6.0] / 4.0 [1.3, 7.3]49.7 [42.0, 57.0] / 50.0 [41.7, 58.3]
RL (DQN-MLP)33.3 [8.3, 58.3] / 8.3 [0.0, 22.2]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]8.0 [2.0, 16.0] / 2.0 [0.0, 5.3]10.7 [3.7, 19.0] / 3.0 [0.0, 7.3]
RL (DQN-CNN)44.4 [19.4, 72.2] / 8.3 [0.0, 22.2]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]9.1 [0.0, 27.3] / 0.0 [0.0, 0.0]12.7 [4.7, 22.0] / 2.0 [0.0, 5.3]16.0 [7.3, 25.3] / 11.3 [6.0, 17.3]
Known-Norm LLM100.0 [100.0, 100.0] / 100.0 [100.0, 100.0]84.8 [63.6, 100.0] / 84.8 [63.6, 100.0]91.7 [77.1, 100.0] / 83.4 [72.9, 93.8]84.8 [69.7, 97.0] / 87.8 [78.8, 97.0]90.7 [83.3, 96.7] / 88.7 [82.0, 94.0]99.7 [99.0, 100.0] / 97.0 [94.3, 99.3]
Norm Oracle100.0 [100.0, 100.0] / 100.0 [100.0, 100.0]100.0 [100.0, 100.0] / 100.0 [100.0, 100.0]100.0 [100.0, 100.0] / 100.0 [100.0, 100.0]100.0 [100.0, 100.0] / 100.0 [100.0, 100.0]100.0 [100.0, 100.0] / 100.0 [100.0, 100.0]100.0 [100.0, 100.0] / 100.0 [100.0, 100.0]
Norm-Blind Planner0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]0.0 [0.0, 0.0] / 0.0 [0.0, 0.0]100.0 [100.0, 100.0] / 100.0 [100.0, 100.0]
Reviewer_xBZwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper tackles the challenge of evaluating how well models can infer previously unknown norms from social interactions. The paper proposes NormGym as a benchmark for studying how well models can perform this task. NormGym is used to, then, show that normative competence is challenging for LLMs, with the highest-performing model achieving a 55.3% success rate. While the community broadly has this intuition already through other existing benchmarks, NormGym contributes a new benchmark and set of new empirical findings to inform and motivate this direction.

Critical strengths

This paper is well-grounded in the literature about social norms and definitions related to norm violations, conventions, and compliance to underlying operative rules in interactions. The strength of this grounding enables the benchmark to study what it says it studies, making it useful for the community of researchers studying social norm adherence in language agents.

The findings related to how agents learn from interactions (the asymmetry in how models use social evidence) are interesting, novel, and useful for the community studying this research challenge. The community has been studying social learning for many years, and this benchmark provides empirical evidence that current models are not good at this form of social learning from other agents.

Critical weaknesses

The following could be viewed as critical weaknesses:

(1) NormGym is a gridworld instantiated by simple interaction settings, which makes it difficult to know how well empirical knowledge gained from NormGym can transfer to more realistic settings. The experiments use 17 held out norms, so it is also difficult to know how robust the conclusions are to the norm selection and training variation.

(2) The benchmark uses an LLM judge to assess whether each agent's natural language norm belief matches the ground truth norm. GPT-5.6 Sol is used as a judge to assign an integer score on a Likert scale from 1 to 7 that represents how well the agent's belief captures the underlying norm. Appendix D2 mentions that 31 participants on Prolific provided human ratings to study LLM judge agreement with humans. However, the paper's methodology leads to vague statements about how well humans agreed with judges. Line 1123 says "Human raters generally agreed with the LLM judge's evaluations" but the humans were providing scores on a Likert scale of 1 to 5 to show how much they agreed with the judge after reading the judge's score. This human evaluation setup does not establish that independent human ratings would align with the LLM judge's scores on a scale of 1-7. A blind human evaluation in which humans actually rated samples on the same 1-7 scale, given the same instructions as the judge, would enable inter-rater agreement metrics to be computed to study the quality of the judge.

The following is not a critical weaknesses, but should be addressed to improve the paper:

The introduction starts by using a robot as an example to motivate the paper, but this paper does not engage at all with the social robotics literature, which has studied normative social robot behavior for decades. It would be good to pick a different example that is more relevant to multi-agent LLM setups.

Questions for authors

Why was joint success human accuracy so low on the allocation norms? Is this benchmark norm category valid for community use? Table 1 shows human accuracy as 67% for the joint success in that norm category. Appendix C4 says that each participant completed two norms with the same interaction and testing interface as the evaluated agents.

What future directions are implied by the findings of this paper? Section 5.2 ends with the statement, "These results suggest that we may need novel learning methods for producing robust normative competence." The paper could benefit from more discussion of possible directions highlighted by findings from this benchmark (especially directions illuminated by this benchmark specifically, that were not made possible by prior benchmarks; this would also help motivate the need for this new benchmark).

6 comments
AuthorsOpenReview ↗

W1 — Real-world generalizability

Thank you for raising this point. Thanks for the opportunity for us to clarify how NormGym connects to real-world settings and why we believe a simplified environment is valuable.

Although the norms in our setting are simple and arbitrary, this is a design choice that we made. We want to test whether agents can actually learn a new norm from social interaction, rather than rely on rules they may already know from pretraining. This ability is important because in real-world settings, it is not possible to tell every rule or expectation to agents since different communities may have their own norms, and many of them are not written down in the rule book. This is why agents need to be able to pick up information from their surroundings and quickly adjust their behavior. Thus we design arbitrary norms that agents would not know already from common sense or prior knowledge from pretraining. We believe this is the best way to evaluate this ability, which is missing from existing benchmarks, as discussed in our introduction in line 52.

At the same time, our four norm families reflect common challenges in the real world when the agent is working with different groups of people who can have their own rules.

  • Access: A coding agent may need to learn that certain files in a shared repository should not be modified.
  • Procedure: A web agent may need to learn that a community requires posts to be reviewed before they are published.
  • Coordination: Multiple coding agents working on the same project may need to learn how to coordinate their changes to avoid interfering with each other's work.
  • Allocation: An agent using a shared computing cluster may need to learn that some GPUs are reserved for other teams, even when they are not being used.

Even though NormGym is just a gridworld setting, it captures the same underlying challenge that agents face in real-world social environments. As the other reviewer mentioned regarding the HuggingFace incidents, the results from our experiments show a similar challenge that current agents lack normative competence, given the fact that they can successfully pursue their own goals but break the rules or expectations of the community around them. The challenge in both cases is that the agents need to understand what is acceptable in the environment they are operating in. We intentionally simplify this problem in NormGym to test this ability. We explicitly tell agents that there is a norm they need to figure out, and we give them clear feedback on whether an action violates the norm, compared to real-world situations where agents may not be aware that there is a social norm and the feedback is often much less clear. However, even when we make the problem this easy, current models still struggle to learn the norms, especially from observing others to understand what an appropriate behavior is. So our intuition is that if agents already struggle under these simplified conditions, we have reason to expect that learning and following community norms would be even more challenging in complex real-world settings. While we do not claim that NormGym directly predicts failures in real-world incidents, we believe these results show a gap in how current agents learn from their social environment.

We see NormGym in a similar way to benchmarks like ARC-AGI-3 [1], AutumnBench [2], and AgentOdyssey [3], which use simplified, game-like environments to test more general agent abilities, such as learning how an unfamiliar environment works through interaction, rather than directly recreating real-world situations. NormGym tests whether agents can infer an unfamiliar social norm through interaction. We take a similar approach to study normative competence, an important ability that has received relatively little attention but will become increasingly important as AI agents operate in shared social environments. We have expanded the discussion of real-world implications in the conclusion of the manuscript.

We have expanded the discussion of real-world implications in the conclusion of the manuscript (lines 498-506).

References [1] ARC Prize Foundation. ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence. arXiv:2603.24621, 2026. [2] Warrier et al. Benchmarking World-Model Learning with Environment-Level Queries. ICML, 2026. [3] Zhang et al. AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents. NeurIPS, 2026.

AuthorsOpenReview ↗

W1 — held-out norms

Regarding the 17 held-out norms, we want to clarify that our main benchmark results are evaluated across all 50 norms (Section 4.4). The 17 held-out norms are only used in our fine-tuning experiments (Section 5), where we randomly split the norms into 33 training and 17 test norms within the four norm families, rather than selecting particular norms. Importantly, our results suggest that the poor performance is not simply due to the choice of held-out norms. As shown in Appendix Table 7, the model actually performs much better on held-out norms after training, with the performance of the Action + Judgment task improving from 39.7% to 70.2% when the models are evaluated in the same training task. However, when we evaluate the trained model using the full interactive NormGym protocol, where it needs to figure out a new norm through interaction, joint success only reaches 13.7% in Table 2. This shows that it does not actually learn how to acquire new norms.

AuthorsOpenReview ↗

W2 — LLM judge validation

Thank you for raising this point. To address this, we conducted a new human evaluation and found that the LLM judge agrees with human raters at the same level that humans agree with each other (Krippendorff's α = 0.73 for both). We also found a strong, statistically significant positive correlation between the judge and human ratings (Spearman ρ = 0.73, permutation p < 0.001).

In this new study, we recruited 30 participants on Prolific. We sampled 100 norm beliefs covering all four norm families and all seven judge score levels. Each participant rated 10 beliefs. Each belief was independently rated by 3 participants, which gives us 300 ratings in total. Participants saw the ground-truth norm and the agent’s norm belief and rated the belief on the same 1-7 scales as the judge, without seeing the judge’s score.

When we averaged the three human ratings for each belief, the judge's correlation with that average rises to ρ = 0.81. These results are a meaningful level of agreement, especially given the nature of the task. Evaluating how well a natural-language belief captures a norm is somewhat subjective. Since each norm is different, this is not a simple checklist where raters can just count how many conditions are met. Even with the same scoring rubric, people may have slightly different opinions on how well a belief captures the true norm. The fact that the LLM judge agrees with humans at a level comparable to how humans agree with each other gives us confidence that its ratings are consistent with human judgments.

We have updated the human-validation results in Section 4.4.2 (lines 407-411) and replaced the previous validation study in Appendix D.2 with this new blind human evaluation, including the agreement and correlation statistics.

Thank you for the suggestion about the Introduction. We have revised the opening paragraph in the introduction.

Reviewer_7F15OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a normative competence nenchmark, which aims to evaluate whether an agent can acquire a community-specific norm it didn't know beore joining the community. The benchmark consists of grid-world environments with predefined norms, and LLM agents can act in them in the ReAct style. The authors conducted experiments across many models as well as three reference arms, and they find frontier models cannot reach human-level performance when they need to infer a hidden norm.

Critical strengths
  • The problem this paper studies is important given increasing deployments of AI agents. Acquiring unknown norms, as opposed to recalling a known one, is also an important research gap that need sto be filled.
  • The operationalizations of norms / violations / sanctions are clean in the grid-world environment.
  • The result that models can't infer indden norm well is interesting.
Critical weaknesses

1. One of the headline findings (models learn from their own violations but not from others') is confounded by the ReAct scaffold (which favors the agent's own sanctions).

According to App C.2: memory records only "an indication of whether the newcomer was sanctioned", while the extra persistent buffer keeps up to 5 of the agent's own sanctioned steps. The prompt adds an explicit {OWN_VIOLATION_NOTE} (see App H). Other agents' sanctions appear only as a red X in the current rendered frame, for one timestep, and are never persisted in text. Others' violations are therefore conveyed only visually and transiently, and need VLM perception of a small marker. This scaffold design might be the cause of the difference between own vs others' violations.

2. Generalizability to real-world multi-agent incidents.

Why does agent behavior in this simplistic grid-world environment generalize to real world multi-agent incidents driven by language models in text-based open environments (e.g. the OpenAI HuggingFace incident)? I agree that evaluating normative competence is helpful but it would be more impactful if this competence translates to some behavior tendancy in real-world scenarios. I don't expect any formal proof but it would be helpful if the authors can clarify the connection.

3. Related work. These two paper seem relevant from the enforcement perspective. I would recommend the authors include them in related work discussion: [1] CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas. [2] Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems.

Questions for authors

See weakness section.

4 comments
AuthorsOpenReview ↗

W1 — Missing information about other agents' violations in the prompt

Thank you for pointing this out. To address this, in the prompt we added an explicit note about other agents' violations everywhere the agent is told about its own (though the other agents’ violations have already appeared in the rendered image):

  1. The current-step prompt now says "In the previous step, [agent name] violated the social norm, so [name] now has a norm-violation marker and is being penalised in this step," next to the existing note on the agent's own violation.
  2. Each step in the 10-step recent memory now lists which other agents violated the norm.
  3. The persistent record of earlier violations now also says which other agent violated the norm, not only the agent's own.

Nothing else changed, and we re-ran GPT-5.6 Sol on all 50 norms.

The same pattern remains after these changes. The table shows the average change in belief correctness immediately after each type of event (same measure as in the paper Figure 4 panel c), reported as mean ± SE. Explicitly including other agents' violations in the prompt and memory increased belief improvement after observed violations from 0.023 to 0.057. However, the model still updated its beliefs more strongly after its own violations (0.162), while belief changes following no violation remained minimal (0.004). This shows that the difficulty of learning from others is not solely due to how violation information was presented in the original prompt. We have added this ablation experiment and its results in Section 4.4.2 (lines 421 to 426) and Appendix D.3, with the modified prompt and memory format provided in Appendix H.1.

Table 1. Effect of explicitly recording other agents’ violations in the memory on belief revision

ConditionOwn violationObserved violationNo violation
Original (data reported in the paper, without other-agent-violation note)0.23 ± 0.100.023 ± 0.0130.005 ± 0.003
New (with other-agent-violation note)0.162 ± 0.0610.057 ± 0.0160.004 ± 0.002

We also checked whether these prompt changes affected the model's overall performance. Joint success remained similar, changing from 44.0% to 48.7% on seen layouts and from 40.7% to 44.0% on unseen layouts. Paired Wilcoxon signed-rank tests across all 50 norms showed no statistically significant difference in joint success between the original run and the new run, on either seen layouts (p = 0.34) or unseen layouts (p = 0.43). Even with the additional information about other agents' violations in the prompt, the model still performs poorly. This gives us additional confidence that our overall findings are not driven solely by how other agents' violations were presented in the original prompt.

AuthorsOpenReview ↗

W2 — Real-world generalizability

Thank you for raising this point, and thanks for the opportunity for us to clarify how NormGym connects to real-world settings and why we believe a simplified environment is valuable.

One difference from incidents like the Hugging Face case is that we focus on whether an agent can learn from the people around it who are already members of a community and adjust its behavior to fit into that community. Although the norms in our setting are simple and arbitrary, this is a design choice that we made. We want to test whether agents can actually learn a new norm from social interaction, rather than rely on rules they may already know from pretraining. This ability is important because in real-world settings, it is not possible to tell every rule or expectation to agents since different communities may have their own norms, and many of them are not written down in the rule book. This is why agents need to be able to pick up information from their surroundings and quickly adjust their behavior. Thus we design arbitrary norms that agents would not know already from common sense or prior knowledge from pretraining. We believe this is the best way to evaluate this ability, which is missing from existing benchmarks, as discussed in our introduction in line 52.

Our norms reflect common challenges in the real world, especially when the agent is working with different groups of people who can have their own rules. These can be different real-world settings, not only physical environments or text-based platforms. For example,

  • a delivery robot entering an unfamiliar community may need to learn that people use one door for entering and another for exiting. Humans can often pick up on these expectations naturally by observing how others behave.
  • Similarly, a coding agent joining a team working on a shared repository may need to learn an unwritten norm that code changes should be tested before submitting them. It could learn this by observing how others submit their changes and how the team responds when someone skips testing.

In these cases, we need the agents to learn from how others behave to understand what is acceptable in that community.

Our setting still shares the same underlying principle as the real-world multi-agent incidents. The results from our experiments and the Hugging Face incidents both show that current agents lack normative competence, given the fact that they can successfully pursue their own goals but break the rules or expectations of the community around them. The challenge in both cases is that the agents need to understand what is acceptable in the environment they are operating in. We intentionally simplify this problem in NormGym to test this ability. We explicitly tell agents that there is a norm they need to figure out, and we give them clear feedback on whether an action violates the norm, compared to real-world situations where agents may not be aware that there is a social norm and the feedback is often much less clear. However, even when we make the problem this easy, current models still struggle to learn the norms, especially from observing others to understand what an appropriate behavior is. So our intuition is that if agents already struggle under these simplified conditions, we have reason to expect that learning and following community norms would be even more challenging in complex real-world settings. While we do not claim that NormGym directly predicts failures in real-world incidents, we believe these results show a gap in how current agents learn from their social environment.

We have expanded the discussion of real-world implications in the conclusion of the manuscript (lines 498-506).

AuthorsOpenReview ↗

W3 — Related work

Thank you for the suggestions. We have added both papers to the discussion of social norms in multi-agent systems in the related work section (lines 133-135).

Thank you again for your time and thoughtful feedback! As the discussion period is coming to an end, we wanted to kindly follow up and ask whether our responses and additional experiments have addressed your concerns. If there are any remaining questions or points you would like us to clarify, we would be very happy to discuss them further!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful feedback and suggestions. We appreciate that the reviewers recognized several strengths of our work, including the importance of studying how agents acquire unfamiliar community norms rather than simply recalling known ones (7F15); the strong grounding in social norms research and clear operationalization of norms, violations, and sanctions (7F15, xBZw); the interesting and novel findings on models' difficulty learning from social interaction, particularly from other agents' experiences (7F15, xBZw); and the value of our human evaluation, Known-Norm baseline, and process-level analysis of belief acquisition (jVQk).

We have carefully considered the concerns raised and conducted additional experiments and analyses to address them. We revised the manuscript accordingly, and all newly added or revised text is highlighted in blue in the revised manuscript. We would like to briefly summarize the main concerns and our responses to them below.

  1. Learning from other agents' violations (Reviewers 7F15 and jVQk). To address the concern that our original prompt made the agent's own violations more salient than others', we ran an additional ablation on GPT-5.6 Sol on all 50 norms, explicitly including other agents' violations in the current-step prompt and memory. The same pattern remains: the model still updated its beliefs more correctly after its own violations than after observing other agents' violations or no violations. We have added these results in Section 4.4.2 (lines 420-425) and Appendix D.3, with the modified prompts in Appendix H.1.
  2. Connection to real-world multi-agent environments (Reviewers 7F15 and xBZw). We clarified how NormGym relates to real-world settings where agents enter new, unfamiliar communities with their own unwritten norms. We provided several real-world examples in our responses to illustrate this connection. We deliberately use arbitrary norms to ensure that agents must learn from social interaction rather than rely on rules they already know. Our results show that current agents struggle with this ability even when we explicitly tell them to infer the norm and provide clear feedback about violations. We have added this discussion in the conclusion (lines 498-506).
  3. Reliability of the LLM judge (Reviewer xBZw). We conducted a new blind human evaluation with 30 participants and 300 independent ratings. We found that agreement between the LLM judge and human raters was comparable to agreement among human raters themselves (Krippendorff's α = 0.73 for both). There is also a statistically significant correlation between the judge and human ratings (Spearman ρ = 0.73, permutation p < 0.001). We have updated Section 4.4.2 (lines 407-410) and Appendix D.2 with the new evaluation and results.
  4. Variability in norm acquisition and confidence intervals (Reviewer jVQk). We clarified that the interactive learning phase consists of 200 continuous steps, during which agents repeatedly observe new situations and can update their beliefs about the norm at each step. To further check the consistency of norm acquisition, we also repeated the learning phase on 20 norms under the same environment conditions. The average final-belief score was 3.65 in both runs, with no significant difference (paired Wilcoxon signed-rank test, p = 0.89). The six test episodes evaluate whether what they learned generalizes to different initial conditions and layouts, rather than simply testing whether it can execute the belief. We have clarified the evaluation protocol in Appendix C.1 and added the repeated-learning results in Appendix D.5. We also computed 95% bootstrap confidence intervals over norms, with the full results added to Appendix D.4.
  5. Future research directions (Reviewer xBZw). We have expanded the Limitations and Future Work discussion in the conclusion (lines 506-518) to highlight three directions motivated by our findings, including training agents to learn from their social surroundings, developing better exploration and belief-updating strategies, and learning to identify and use relevant social information efficiently.

We hope these additional experiments, clarifications, and manuscript revisions address the reviewers' main concerns and help strengthen the paper. We sincerely appreciate the time and effort the reviewers and Area Chair have put into evaluating our work. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.