- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
This paper investigates whether a newcomer agent can identify the shared hidden norm of existing agents in a multi-agent interaction. To address this question, the authors introduce a community debate setting and design experiments involving different normative modules to quantitatively evaluate and validate the agents' ability to identify these norms. Additionally, the paper introduces overimitation experiments in which background agents exhibit idiosyncratic behaviors. The authors also conduct costly overimitation experiments, using a token budget to force LLMs to select the community norm over other behaviors. Finally, the paper claims that LLMs merely imitate observed behaviors rather than truly learning the hidden norms.
Critical strengths
- The proposed problem of hidden norms is interesting.
- Designing silly norms that are absent from the model's pretraining data to prevent it from relying on prior knowledge is a reasonable approach.
Critical weaknesses
- The proposed research problem is not novel, as previous studies have already explored similar questions [1], including requiring models to follow norms. Even the name of the proposed method is the same. However, this paper provides neither relevant comparisons nor citations. Furthermore, the related work discussion mainly focuses on alignment and social aspects, which are less directly relevant to this paper. In fact, several studies have already explored agentic approaches to addressing such sociological problems in LLMs [2].
- Describing the different normative modules solely through text is too abstract. I suggest providing a diagram that directly illustrates the specific differences between the modules.
- Regarding the over-imitation problem emphasized in this paper, the experimental design is insufficient to support the claim that models merely imitate behaviors, as discussed in Question 2. Moreover, the proposed normative reasoning framework to be more of an in-context engineering approach. Overall, the contributions of this paper are quite limited.
- The paper claims to use frontier-level models. Setting aside model versions, GPT-5 Nano, Qwen3 8B, and Qwen3 32B are not even among the stronger models within their respective model families. Therefore, describing them as "frontier-level" is an overstatement.
- The authors should not put all ablation studies in the appendix. Ablation studies are essential experiments, and their key comparisons should be presented in the main paper, with the appendix only for additional details. Furthermore, several important ablation studies are missing, such as examining the impact of the presence or absence of sanctions in Multi-Agent Debate and the effect of different token budget sizes in costly overimitation experiment.
- Evaluating the proposed approach on only one dataset (MMLU-Pro) limits the credibility of the experimental results.
- Regarding the writing, the paper lacks a clear conclusion. The Discussion section mainly focuses on potential future research directions without clearly summarizing the key findings of this work. The authors should explicitly explain the core value of this paper.
References:
[1] Sarkar, Atrisha, et al. "Normative modules: A generative agent architecture for learning norms that supports multi-agent cooperation." arXiv preprint arXiv:2405.19328 (2024).
[2] Vinitsky, Eugene, et al. "A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings." Collective Intelligence 2.2 (2023): 26339137231162025.
Questions for authors
- Regarding Weakness 1, could the authors explain more specifically the differences from previous studies?
- Using a token budget to penalize overimitation and then claim that LLMs cannot distinguish between norms and idiosyncratic behaviors is not a reasonable inference. A decline in norm-following performance under a token limit does not imply that the model cannot correctly learn the norm. This deduction is too indirect. Why not directly ask the newcomer agent which behaviors it considers important and then analyze the probability distribution of importance across norms and idiosyncratic behaviors? Wouldn't this provide a more direct way to understand what the model actually focuses on?
8 comments
Regarding the over-imitation problem emphasized in this paper, the experimental design is insufficient to support the claim that models merely imitate behaviors, as discussed in Question 2. Moreover, the proposed normative reasoning framework to be more of an in-context engineering approach. Overall, the contributions of this paper are quite limited.
We thank the reviewer for this comment. We agree that the normative modules can be viewed as in-context scaffolds: they are a harness that gives a capable model the structure it needs to attend to norms, and the paper compares four ways of providing this structure. They are more than prompting on the newcomer’s own model because the module calls a separate, possibly more powerful model to do the norm-focused work (§4, Fig. 1), but we do not claim them as the paper’s main contribution. The contribution is the controlled testbed and the finding about what such scaffolds can and cannot do: even a harness built to infer norms from sanctions fails to separate the enforced norm from idiosyncratic noise. On over-imitation, we agree that “merely imitate” is stronger than our evidence strictly supports, and we have softened the wording in the abstract and discussion accordingly. The evidence for indiscriminate copying comes from several angles: over-imitation rises with norm-following ability (Fig. 4), inferred norm descriptions themselves include the idiosyncratic behaviors (Fig. 4, §A.4), description accuracy does not predict reduced copying (Fig. 6), and making imitation costly causes accuracy and norm-following to collapse rather than making models more selective (Fig. 5). We have also added a figure of the setup and normative modules to better illustrate their design as mentioned in our response to your previous comment above.
The paper claims to use frontier-level models. Setting aside model versions, GPT-5 Nano, Qwen3 8B, and Qwen3 32B are not even among the stronger models within their respective model families. Therefore, describing them as "frontier-level" is an overstatement.
We thank the reviewer for pointing this out. Our use of “frontier-level” models referred to Claude Sonnet, which was used to evaluate the baseline with a stronger model than was used in our main experiments (Introduction, “We demonstrate that a memory-only baseline (even powered by frontier models) consistently fails to learn novel norms”); we will make this more clear in the paper. Beyond the baselines, we agree that “frontier-level models” may not accurately reflect the capabilities of the models used in our experiments, and we will replace it with more precise language describing the specific models evaluated.
The authors should not put all ablation studies in the appendix. Ablation studies are essential experiments, and their key comparisons should be presented in the main paper, with the appendix only for additional details.
We appreciate this comment, and would like to refer the reviewer to our response to reviewer 7a54, in which we outline the key contributions and results and where they are located in the main text of our paper. We would welcome a discussion of any specific material beyond what we have outlined that the reviewer believes should appear in the main text.
Furthermore, several important ablation studies are missing, such as examining the impact of the presence or absence of sanctions in Multi-Agent Debate and the effect of different token budget sizes in costly overimitation experiment.
We appreciate the reviewer’s suggestion, and are running additional ablation experiments to address your comments, which we will incorporate in the final version of our paper when complete.
Presence or absence of sanctions. In our setting, a norm is by definition a behavior whose absence the community sanctions. Removing sanctions therefore removes the norm, and the setup reduces to standard multi-agent debate, which has been studied extensively [6,7]. Such a condition has no norm to acquire, so it cannot speak to normative competence, which is the object of our study.
Different token budget sizes. We are currently running these ablations on a representative subset of our norms, under a token budget of 100, 200, 300, or 500 words. Based on our initial results thus far, we observe the same general trend that models see a reduction in performance and norm following, but with a smaller token budget corresponding to a greater decrease in performance and norm following. This indicates that we continue to see similar results even under different (and much more lenient) costs imposed on the newcomer for overimitation.
We will update the paper with the final results from these ablations when experiments are complete.
Evaluating the proposed approach on only one dataset (MMLU-Pro) limits the credibility of the experimental results.
In our design, the dataset serves as the vehicle for debate, and the capacity we study is social and normative competence, not task performance. The task only determines what the community talks about, and the norm determines whether the newcomer is allowed to hear it. We therefore held the task fixed (MMLU-Pro, chosen for its reasoning-heavy problems that benefit from debate; §5) so that we could vary the things that matter for our research question: norms, normative module designs, models, idiosyncratic behaviors, and costs on imitation. Adding datasets would mostly tell us how debate behaves on different kinds of problems, which is an interesting but separate question from whether agents can discern an enforced norm. Norm-following is also measured directly from the form of the newcomer’s messages, independent of question content (Fig. 2-3). The task matters only through the accuracy benefit of access to debate (Fig. 1b), which gives the newcomer its incentive to learn the norm. The one experiment where task and norm could interact is the token-budget penalty (§5.3), so we ran an ablation varying the word limit between 100 and 500 words; see our response above. We agree that this limitation should be stated clearly, and in the revision we will explain this choice in §5, add the word-limit ablation to the appendix, and list other tasks under limitations and future work in §7.
Regarding the writing, the paper lacks a clear conclusion. The Discussion section mainly focuses on potential future research directions without clearly summarizing the key findings of this work. The authors should explicitly explain the core value of this paper.
We thank the reviewer for this suggestion. Our Discussion section already states the paper’s central finding (“there is a dissociation between two capacities - behavioral imitation and normative competence - that current approaches cannot pull apart”), but we agree it could be clearer. Behavioral imitation and normative competence are two distinct capabilities that are easy to conflate. Models can look as if they are learning norms because they are good at imitating observed behavior, but imitation does not require identifying which behaviors the community actually enforces. Our results show that the two are separate capabilities: norm-following success is largely tied to imitation ability rather than to genuine normative competence, since models that follow the norm better also copy more idiosyncratic behavior (Fig. 4), and they do not become more selective when copying is costly (Fig. 5). The core value of the paper is therefore to identify this as a measurable capability gap, with implications for agents entering communities whose norms differ from their training data. In the revision, we have rewritten the opening of the Discussion to state this finding and its value more clearly.
Regarding W1, could the authors explain more specifically the differences from previous studies?
We appreciate this feedback from the reviewer to improve our paper. Please see our responses to W1 above, as well as the associated edits made in the revised paper.
We sincerely appreciate your thoughtful feedback. We also thank you for highlighting that our work tackles an “interesting problem” and the “design of silly norms absent from pretraining data” in our setting as a strength of our approach.
Below we respond to your comments:
We appreciate this point from the reviewer to cite this relevant work, and we have added a reference to this work addressing it in our revised paper (§6). We borrowed the term “normative module” but not the full framework, which led to the confusion here. Our work shares the motivation and newcomer-and-community framing of [1], but studies a different problem:
Our main findings therefore have no counterpart in [1].
We thank the reviewer for this helpful suggestion, and we agree that we should introduce more references related to sociological problems in LLMs in our related works section. As for [2], we would like to clarify that this paper is about MARL (Multi-Agent Reinforcement Learning), rather than LLMs, and is focused on agents learning norms in order to learn sanctioning behavior (not on how to apply the norms themselves). The comparable element to our work on “normative modules” is that the model learns a classifier (punishable vs acceptable behavior) based on observing what others punish, and there is an implicit reward for punishing what other community members also punish - which is another aspect of normative competence that is distinct but complementary to our work. However, learning the ability to determine when to punish behaviors is not the same as learning to generate a compliant response. More recent work explores norm emergence in generative agent societies [4] and evaluates whether embodied agents comply with unspoken norms [3].
We have added discussion of these and other relevant sources in our revised draft, and appreciate the reviewer’s feedback on this point.
We appreciate the reviewer’s suggestion to improve our paper, and have added visualizations to our draft in Fig. 1 (direct edit) and the appendix, with references in the main text following the updated descriptions. This is in addition to the revisions we call out in our response to reviewer 7a54 who also encouraged us to add more clarity in the main text about our setting and how the normative modules function.