acceptodds
Under review as a conference paper at ICLR 2027

PACT: Emergent Institutions in Long-Horizon Multi-Agent Coding Systems

36 views

Abstract

As autonomous agents become more widely deployed, coordinating access to shared computing resources and services will become increasingly important. Recent incidents have already revealed unplanned coordination and resource sharing between agents, highlighting the safety implications of these interactions. Yet prior studies of agent negotiation often use stylized games and simulated bargaining, leaving open how these behaviors develop during sustained, long-horizon programming work. We propose PACT Programming Agents Contending for Time a framework in which agents solve sequences of programming tasks under individual deadlines while negotiating access to a shared execution resource. Their agreements determine when and for how long they can inspect programs, compile code, and run tests. Furthermore, we develop a 21-category taxonomy of negotiation strategies and emergent norms and use it to analyze eight models and audit more than 1,500 trajectories across different environmental settings. We find that agents spontaneously develop informal institutions: persistent rules and commitments grounded in reputation, reciprocity, and precedent. These arrangements carry across tasks and influence resource allocation, with agents sometimes foregoing immediate advantages to honor earlier agreements. Among the models tested, newer frontier models exhibit more relational behavior than older or lightweight models. Alongside cooperation, we observe exclusionary coalitions and rare divide-and-conquer strategies. PACT provides a controlled setting for proactively and systematically auditing how these relationships emerge, shape access to resources, and affect the safety and coordination of multi-agent systems.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-agent systems, coding agents, long-horizon agents, negotiation, collusion and coalition

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jWpKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines how multiple programming agents repeatedly compete for shared execution resources and how they negotiate to form reciprocal, reputational, alliance, and cross-task-based resource allocation norms.

Critical strengths

It places agent negotiation in a continuous working scenario with real code execution constraints and studies coordinating behavior by adjusting budgets, deadlines, and memory.

Critical weaknesses

This paper lacks some discussion with previous work [1], and I believe many of the risks discussed in this paper are already included in the previous work, which limits the novelty of this paper. Furthermore, the biggest problem with the experimental part is the limited number of models tested; the authors need to add more models to verify their universality and the differences in performance between different models.

In addition, the authors used multiple LLM referees and some human checks, but did not clearly report the human sample size, category-level accuracy, referee consistency, and disagreement handling methods. Some round-by-round trends also come from regular expression heuristics, especially rare deception or exclusionary behaviors, which require stronger human verification.

There is also a design problem: in this paper, the environment artificially lowers the negotiation cost, which may amplify institutional formation. The deadline is paused during negotiation, while queuing consumes time; the authors explicitly state that this setting is to prevent agents from escaping negotiation. Additionally, each execution window has an upper limit to increase the number of negotiation rounds, so the observed continuous negotiation behavior may be highly dependent on this incentive design.

[1] Huang, Yue, et al. "Emergent Social Intelligence Risks in Generative Multi-Agent Systems." arXiv preprint arXiv:2603.27771 (2026).

Questions for authors

see weakness

2 comments

Weakness 1

On novelty relative to Huang et al. (2026). Huang et al. make an important contribution by cataloguing 15 emergent risks, each in a short simulation built for that risk. Their compute-scheduling setting, for example, has three agents, simulated time blocks instead of real work, and one strategic action, a GUARANTEE mechanism whose rules and fees the experimenters fix. PACT asks a different question. Rather than whether a given risk can appear, it asks how agents build their own institutions during real long-horizon work, and what makes those institutions form, persist or fail. This difference shows up in four ways. First, agents in PACT earn their payoffs from real progress on ProgramBench, so negotiation is coupled to actual programming work instead of being the task itself. Second, we provide no allocation rules beyond bidding, so agents write their own rules in language, cite them as precedent and carry them across tasks. Third, we identify the conditions behind these institutions, showing that they require cross-task memory (Table 2)\textcolor{blue}{\text{(Table 2)}}, are shaped by budget inequality (Tables 5 and 6)\textcolor{blue}{\text{(Tables 5 and 6)}} and are not induced by more reasoning (Table 16)\textcolor{blue}{\text{(Table 16)}}. Fourth, while Huang et al. describe risks by analogy to failures in human societies, we ground our analysis in institutional economics and bargaining theory. We define institutions following North (1990), and each of our 21 categories is tied to an established construct, such as reputation (Kreps and Wilson, 1982), reciprocity in repeated games (Mailath and Samuelson, 2015), coalition formation (Ray and Vohra, 1999) and conventions (Young, 1993). This lets us check our findings against theory rather than only report that a behavior appeared. For example, repeated-game theory predicts that reciprocity and reputation need an expectation of future interaction, and these behaviors indeed disappear when memory does not persist. Behaviors like exclusion can look similar on the surface, but our contribution is to show how they arise from agent-made institutions over long horizons, which a single-mechanism, short-horizon setting cannot capture.

We evaluate eight models from four developers in the main analysis (Table 1)\textcolor{blue}{\text{(Table 1)}}. These span open- and closed-weight models and both frontier and efficiency tiers, and Table 15\textcolor{blue}{\text{Table 15}} adds DeepSeek and GPT-5.4. The set also includes controlled comparisons within families (Kimi-K2.6 to K3 and GLM-5.2 to 5.3) and across reasoning effort (Table 16)\textcolor{blue}{\text{(Table 16)}}. We have also added results using OpenCode and report it in Table 8 in Appendix B. We observe similar institutional behavior emerging showing the coding agent harness and its scaffolding has minimal impact

These comparisons show that institutional behavior is not explained by model size or reasoning budget alone. Each run involves five agents negotiating across four long-horizon coding tasks, and our analysis covers over 1,500 trajectories. Some of our ablations also included DeepSeek models. We do not include Gemini models as we during rebuttals have added them as an external judge to reduce self preferential bias. We also want to keep the framework economical as understanding the swarm behavior is crucial and the academic community may not have the means to scale their research to 100 or 1000s of agents.

Weakness 2

For human annotation, we use five specialists: one economist, one data analyst, one linguist, and two computer scientists. We ask them to annotate 100 traces. Because our LLM-as-judge approach is conservative i.e. judges don’t count any labels they are not confident about, we observe very high agreement between the human annotators and the judges. We report the results in Table 10 in Appendix B\textcolor{blue}{\text{Table 10 in Appendix B}}. The disagreements are resolved by considering the disagreements as absence of that behavior. This way we prevent overreporting certain behaviors.
We also check for self-preference, by comparing our judge (Fable) against Gemini Argon, which comes from a model family outside our testing suite. The two judges give exactly the same label 85.9% of the time, and they are within one label of each other 91.3% of the time. So most disagreements are near-misses, not opposite calls. The cell-level correlation is moderate (Pearson r = 0.52), mainly because rare codes are sparse and a single disagreement moves the number a lot. Agreement is stronger at the aggregate level, which is what our analysis relies on. The two judges rank run-and-code totals similarly (Spearman ρ = 0.66), and they agree on how often each of the 22 codes occurs (Pearson r = 0.63) and on which behaviors are common or rare (Spearman ρ = 0.70). Argon assigns about 23% fewer labels than Fable overall (a ratio of 0.77), but it follows the same pattern. This suggests the relative findings hold regardless of which judge we use. We report these results in Table 9 in Appendix B\textcolor{blue}{\text{Table 9 in Appendix B}}. We will add a round level judge for Figure 14\textcolor{blue}{\text{Figure 14}} in camera ready version.

Weakness 3

We don’t think these design choices explain our results. Every model runs under the same paused deadline and the same grant ceiling, yet relational behavior ranges from 0 to 79 of rounds (Table 1)\textcolor{blue}{\text{(Table 1)}}. The efficiency-tier models and Kimi-K2.6 show none at all. Even within a single model, turning off cross-task memory makes relational behavior disappear while the incentives stay the same (Table 2)\textcolor{blue}{\text{(Table 2)}}. So cheap negotiation on its own doesn’t create institutions. The grant ceiling doesn’t either. If more rounds were producing institutions, we would see them build up over a run, but they actually peak early and then fade (Figure 8)\textcolor{blue}{\text{(Figure 8)}}. Agents also aren’t forced to negotiate, since private messages and withdrawal are both optional. We think the setup is realistic. In real systems, agents usually coordinate over a separate channel that doesn’t use up the scarce resource, and schedulers commonly cap how long a job can run. The pressure is still real, because time spent waiting in the queue counts against the deadline. That said, we will add ablations that count negotiation time toward the deadline and remove the grant ceiling.

Reviewer_AXm4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PACT (Programming Agents Contending for Time), an environment for studying multiple coding agents solving programming tasks from ProgramBench under private deadlines and a sing shared execution resource. The paper derives a taxonomy of the negotiation strategies and norms that emerge in 1,500 trajectories from 8 models. The paper claims agents develop emergent institutions.

Critical strengths
  • The paper is easy-to-follow, and it is deeply motived from the real world scenario.
  • The author explains their rationale about choosing the ProgramBench as the base task bench to build upon. The development of the cooperation evaluation environment is also deliberate and intentional. For example, negotiation time is excluded from the deadline because including it led agents to treat negotiation as a “negotiation tax” and skip it.
Critical weaknesses
  • Lack of discussion of task difficulty and model capabilities. Task difficulty is a configurable factor, but the main text never reports which tasks were assigned or how difficult they were. Task outcomes appear only in Appendix Figure 14 (p. 23), stating that “Completion is a flat 0% for every model since ProgramBench is near-impossible in a single subrun.” The paper also does not discuss how differences in the models’ coding ability relate to their negotiation behavior. It would be valuable to see the relation between negotiation behavior to each model’s coding progress.
  • Limited technical depth. Most of the trajectory analysis focus on the frequency counts of taxonomy categories, assigned by LLM judges. Explanations for the observed dynamics are offered as hypotheses drawn from the traces, and are not tested further.

Format:

  • Sec. 3.4, p. 5: the protocol steps are numbered (i) to (iii) and then “Fourth.”
  • Sec. 4, p. 5: “how how” appears, and “21 categories taxonomy of” is ungrammatical.
  • Table 8 caption, p. 21: “reasoning effor.”
Questions for authors
  • What role does the declared position play? Each agent declares a desired position during the bidding round. It is reasonable to assume that all agents would request the first slot. In the appendix examples, all three agents declare “identical 120s first-exploration windows at position 1.” How often do agents request a position other than 1? Can the authors show that the position field affects outcomes?
  • How do negotiated schedules compare with the default scheduler? When agents skip negotiation, allocation falls back to the scheduler’s default, which “can lead to inefficient scheduling or resource starvation.” However, the paper reports no comparison against this default.
  • What do the findings imply for coordinating real agent deployments? The conclusion says the results point to both coordination benefits and risks such as exclusion. It would be interesting to see how coordination improve the task performance metrics.
2 comments

We thank the reviewer for their review and address their concerns below

Weakness 1

In our experiments with 5 agents 2 agents solve easy tasks whereas 3 solve medium tasks. Since it is an easier task. We observe that the agents that have easier tasks require shorter windows, hence they participate in a higher number of negotiation rounds (29% compared to 19% for medium difficulty) early on. This leads to easy challenge solving agents to frontload relational behavior and overall show a higher percentage of relational institution formation. Results are reported in Table 13 in Appendix B\textcolor{blue}{\text{Table 13 in Appendix B}}.

To relate negotiation to coding progress, we compare Kimi-k3 and GLM-5.3. GLM-5.3 makes more progress on the reconstruction task (7.1% vs. 6.2%), and it also negotiates more, taking part in more rounds of conversation and coordinating more with other agents. Coordination doesn’t come free, though. GLM-5.3 misses its deadline about twice as often as Kimi-k3 (65% vs. 32%), even though negotiation time doesn’t count toward the deadline. So the extra misses come from how it uses its execution time, not from time spent talking. Better coordination seems to help agents get more done, but it doesn’t make them better at budgeting their time.

Weakness 2

We would like to clarify the goal of the paper, since it shapes how the analysis should be judged. PACT is not designed to explain each behavior through a single causal mechanism. It is designed to audit how what agents say during negotiation relates to what they actually do with the shared resource. Each part of the analysis pairs a judge-coded behavior with an action recorded in the scheduler logs.

Relational behavior peaks early and declines as mean bids escalate, so pacts give way to bidding wars (Figure 8)\textcolor{blue}{\text{(Figure 8)}}. Models that use the private channel more also show more relational behavior (Figure 5)\textcolor{blue}{\text{(Figure 5)}}. Urgent agents first bid more and request more time, then bid less and request less in later subruns (Table 4)\textcolor{blue}{\text{(Table 4)}}. Richer agents convert their budget into wins only temporarily in GLM-5.3 but durably in gpt-5.6-luna (Tables 5 and 6)\textcolor{blue}{\text{(Tables 5 and 6)}}. Several categories are also defined by actions rather than words. Coalition loyalty requires an agent to give up a better slot in its ballot, and an exclusionary coalition requires a slot to go to a favored agent over rivals with similar claims. The frequency counts are therefore one half of a paired measurement, and the other half comes from logged bids, ballots and allocations. Where a causal claim matters most, we test it directly with ablations of memory (Table 2)\textcolor{blue}{\text{(Table 2)}}, bid mechanism (Table 11)\textcolor{blue}{\text{(Table 11)}} and reasoning effort (Table 16)\textcolor{blue}{\text{(Table 16)}}. We agree the remaining links are correlational, and we will state this framing explicitly in the revision.

Questions For Authors:

Question 1

The scheduler is deterministic and the position declared and included in the vote is the primary factor governing the outcome. The agents declare a position. During negotiation either they keep their declaration or they change their declaration. Their declaration is not just their position. It's the order they think is suitable for every agent and the time they think they should have on the cluster. Since we use Borda aggregation [1], it's not the winner takes all. Each worker submits a full ranking of all workers. A worker ranked i-th on an n-worker ballot gets n − i points, and the scheduler runs workers in order of their total points across all ballots.

The agents don’t keep requesting the first position because they realize that other positions also have value and a “fractured vote” (in agents language) takes away the chance of getting an ideal non-first position. The agents only request the 1st position 27% of the time. The agents declaring the first position and supporting it with the highest bid win the position more than 90% of the time.

Question 2

We study strict bid based default scheduler ordering by disabling negotiation and only relying on declared bids and position order. The lack of institutions or pacts that arise from negotiation can lead to increased unfairness and shorter windows. The results are reported in Table 11 in Appendix B\textcolor{blue}{\text{Table 11 in Appendix B}}.

We also notice that since there is increased understanding between agents they reduce the position 1 ask and reach 2-3 as their main ask for faster alignment.

Question 3

On task performance, coordination helps progress but does not by itself make agents use their time better. Among the frontier models, GLM-5.3 coordinates the most, using the private channel in 74.5% of rounds, and it also makes the most progress (7.1% vs. 6.2% for Kimi-k3). However, it misses its deadline about twice as often as Kimi-k3 (65% vs. 32%). GLM-5.2 shows that coordination alone is not enough, since it is highly relational but makes only 4.0% progress. These results can be found in Figure 5 and Figure 16 in Appendix B\textcolor{blue}{\text{Figure 5 and Figure 16 in Appendix B}}.

For real deployments, our findings point to four practical levers. First, memory decides whether institutions form at all, since relational behavior disappears without cross-task memory (Table 2)\textcolor{blue}{\text{(Table 2)}}, so operators effectively choose whether agents build their own rules. Second, these institutions are fragile and give way to bidding wars as prices rise (Figure 8)\textcolor{blue}{\text{(Figure 8)}}, so they should not replace explicit allocation guarantees. Third, emergent coordination can be unfair, as exclusionary coalitions favor in-group agents and a richer gpt-5.6-luna agent keeps winning even when depleted (Table 6)\textcolor{blue}{\text{(Table 6)}}, so allocations need auditing. Fourth, institutional behavior depends strongly on the model (Table 1)\textcolor{blue}{\text{(Table 1)}}, so mixed-model deployments may disadvantage cheaper agents that do not take part in these arrangements.

References\textbf{References}

[1] Dwork, Cynthia, et al. "Rank aggregation methods for the web." Proceedings of the 10th international conference on World Wide Web. 2001.

Reviewer_jEprOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on the problem of coordinating access to a shared resource among a group of automated agents attempting to solve tasks individually. The authors are particularly interested in studying and evaluating the role and implications of negotiation among the coordinating agents in this process. To this end, they introduce PACT, an environment in which LLM coding agents rebuild ProgramBench programs under private deadlines and budgets. The agents share a single execution resource and need to procure an execution time window to inspect, compile or test their program. Time is allocated via repeated negotiation rounds in which the agents declare their current status, their bids and cast ballots that determine the schedule. The prompts instruct agents to treat the bid as the primary signal and for various negotiation mechanisms to be viable, the agents need memory to be carried across different tasks. The authors derive a taxonomy of 21 negotiation strategies and norms from interaction trajectories. They further label negotiation rounds from eight models with LLM judges. Empirical results demonstrate that frontier models develop informal instituions in the form of rules that persist across tasks and commitments grounded in reciprocity, reputation and precedent. The authors also find agents forming coalition aimed at excluding other agents and divide and conquer strategies adopted by agents in few cases. Most of these results do not hold for smaller and older models which eventually fall back on conventional heuristics such as shortest job first.The authors also explore various experimental setups by varying budgets, deadlines, bid updates and reasoning effort and audit more than 1,500 trajectories across these settings.

Critical strengths

I find the testbed setup to be quite strong with solid coverage of properties useful for studying how conventions form among deployed agents. The authors have taken great care in designing the environment such that it produces the phenomenon of interest such as rule formation or costly compliance, enabling the analysis of emergent behaviors. The most critical aspect of the environment is that it binds negotiations with the actual task objective as the agents cannot make progress in their individual task without engaging in the bargaining process for execution time. This reasonably addresses the limitation of bargaining benchmarks and stylized social dilemma games where the interaction is the task and the payoffs are fixed by the designer. Also while the taxonomy is extensively studied in economics and social science literature making it is not novel per se, I do find its derivation useful in the context of the setup to precisely analyze various aspects of the emergent behavior.

Critical weaknesses

My two biggest concerns revolve around issues in the overall measurement process and the resulting claims:



  • It is very hard for me to figure out whether the observed phenomena are truly emergent or the artifact of the scaffolding. For instance, Section 4 and Appendix D seem to reveal confounding factors. Specifically, in Section 4, authors state that agents create, cite and enforce conventions without explicit instructions but prompts in Appendix D steer toward the behaviors being counted. Further, the history summarizer is told to preserve conditional commitments, references to earlier rounds, and any rule the group settled on for breaking ties. It appears that the rise in rule citation across tasks (Table 3) is consistent with a summarizer that keeps what it was told to keep so how do we separate emergence of behavior from just effective retention of memory? Prompt phrasing alone can increase collusion among LLM pricing agents (Fish et al., 2024), so sensitivity of this kind should be expected and tested. This weakness matters a lot for my assessment as emergence is the central claim of the paper and the current design cannot rule out the simpler explanation.

  • Another claim that paper makes quite strongly is that informal institutions that emerge and conventions shape later allocations which I find weakly supported at best. Every quantitative result rests on LLM judge labels whose reliability is not reported. Section 5.1 announces three judges that include, at least two from families that are also under evaluation. It invites self-preference (Panickssery et al., 2024). The human verification has no sample size or agreement statistic and Figure 12 falls back on regular expressions. Further, the paper fails to report how often the schedule departs from bid order or how often commitments are kept. Tables 5 to 7 report win rates and how concentrated wins are, but nothing ties those numbers to commitments or rules. Unfortunately I find the safety framing in the earlier parts of the paper not really tested in experiments for similar reasons. For instance, the authors cite the recent work (Paglieri et al., 2026) on emergence of cheating and whistleblowing in agent populations solving mathematical problems but never analyze the trajectories for type of enforcement or institutional behaviors reported by that paper.

[1] Algorithmic Collusion by Large Language Models, Fish, Gonczarowski and Shorrer 2024

[2] LLM Evaluators Recognize and Favor Their Own Generations, Panickssery, Bowman, Feng, 2024

Questions for authors
  • How are ballots aggregated into the final schedule? The negotiator prompt says no fixed clearing rule applies but then one agent expects the outcome to land via aggregate? I believe all the expected phenomena to emerge have direct connection on how this is setup so this needs to be clarified.
  • Was the history summarizer of Appendix D active in every memory condition in Tables 2 and 3? What would happen if one were to replace it by raw transcripts?
  • How were disagreements resolved among judges and what agreement did the judges reach with each other and with the human annotators?
  • How many runs, workers per run and rounds underlie each entry of Tables 1 to 5 and how much do the entries vary across runs?
  • There are few questions on the commitment analysis itself that need to be reported. For instance how often does the final schedule depart from bid order and what share of those departures follows a prior commitment or a cited rule? How often are commitments broken? Any implications of broken commitments?
  • I was not able to see an experiment with mixed models. Am I missing it? If not, what happens to the reported results in mixed populations of different models?
  • How do the emergent institutions compare with designed allocation rules for instance, strict ordering by bid or earliest deadline first on the distribution of access?
  • Can the taxonomy distinguish shared strategies, norms, and rules in the sense of Crawford and Ostrom (1995). Their literature separates norms and rules vs shared strategies by whether a statement carries a prescription and a sanction? The sanction in particular is what distinguishes a rule from a shared habit and the current categories seem to lack the record of it.
3 comments

We thank the reviewer for their review and address their concerns below

Weakness 1

We would like to clarify that the summarizer does not prompt agents to form relationships. It only condenses older negotiation history and keeps what agents themselves already said, so that this information is not lost as the history grows. It also includes system-level information to keep agents from hallucinating architecture details. The summarizer cannot explain our main results, since Tables 1 and 2\textcolor{blue}{\text{Tables 1 and 2}} report results from a single subrun per run, where the summarizer is not used (we will clarify this in the paper). In addition, every model uses the same summarizer’s prompt, yet the efficiency-tier models show no relational behavior at all (Table 1)\textcolor{blue}{\text{(Table 1)}}. Finally, even though commitments are kept in the digest, relational behavior fades over a run and agents move into bidding wars (Figure 8)\textcolor{blue}{\text{(Figure 8)}}.

Table 3\textcolor{blue}{\text{Table 3}} (rule citation) uses the summarizer, and we agree the summarizer affects its trend. To isolate this confound, we split rule citation into two groups. The first covers citations of time and urgency rules, which grow as deadlines approach and agents request larger windows. The second covers the creation and citation of institutional rules. The rising trend in Table 3\textcolor{blue}{\text{Table 3}} comes entirely from the first group. Standard rule creation and citation stays similar whether the summarizer is on or off, and it actually decreases over subruns (Table 7 in Appendix B)\textcolor{blue}{\text{(Table 7 in Appendix B)}}. However, time-based arguments increase compared to S1, which also leads to an increase in requested time with the summarizer both enabled and disabled. The increase therefore reflects growing deadline pressure and requested time, not the summarizer creating or reinforcing institutions. We have reported the time results Figure 9 and Figure 10 in Appendix B\textcolor{blue}{\text{Figure 9 and Figure 10 in Appendix B}}. While the prompt phrasing can have an impact, we intentionally made sure that the prompts do not mention institutions, collusion, coalitions, or any idea of building relationships across multiple steps. The prompt is optimized for the primary goal of agents negotiating to get the optimal slot. We also intentionally do not optimize for institutional behavior, as we were not expecting to observe it to this degree, which makes it truly emergent. This gives future work the opportunity to optimize prompts to further induce or mitigate institutional behavior depending on the task, and to understand its impact on performance. For the remaining scaffolding (i.e., the coding agent harness), we use the out-of-the-box Pi harness with no modifications for the ProgramBench task. To further show that the behavior is not specific to the coding agent scaffolding or harness, we also run it on OpenCode and report it in Table 8 in Appendix B\textcolor{blue}{\text{Table 8 in Appendix B}}. We observe similar institutional behavior emerging showing the coding agent harness and its scaffolding has minimal impact.

Weakness 2

The primary goal of the paper is to connect the language agents use with the actions they take while forming these institutions. To do so, we do not rely only on LLM-as-a-judge analysis. We tie it closely to the number of channels created and the bids placed, and we have collected a new set of results on time requested (reported in Figure 9 and 10 in Appendix B\textcolor{blue}{\text{Figure 9 and 10 in Appendix B}}). To correct for self-preference bias, we also verify our judge results with Gemini Argon, a model from a family that is not part of our testing suite.

To check for self-preference, we compare our judge (Fable) against Gemini Argon, which comes from a model family outside our testing suite. The two judges give exactly the same label 85.9% of the time, and they are within one label of each other 91.3% of the time. So most disagreements are near-misses, not opposite calls. The cell-level correlation is moderate (Pearson r = 0.52), mainly because rare codes are sparse and a single disagreement moves the number a lot. Agreement is stronger at the aggregate level, which is what our analysis relies on. The two judges rank run-and-code totals similarly (Spearman ρ = 0.66), and they agree on how often each of the 22 codes occurs (Pearson r = 0.63) and on which behaviors are common or rare (Spearman ρ = 0.70). Argon assigns about 23% fewer labels than Fable overall (a ratio of 0.77), but it follows the same pattern. This suggests the relative findings hold regardless of which judge we use. We report these results in Table 9 in Appendix B\textcolor{blue}{\text{Table 9 in Appendix B}}

For human annotation, we use five specialists: one economist, one data analyst, one linguist, and two computer scientists. We ask them to annotate 100 traces. Because our LLM-as-judge approach is conservative i.e. judges don’t count any labels they are not confident about, we observe very high agreement between the human annotators and the judges. We report the results in Table 10 in Appendix B\textcolor{blue}{\text{Table 10 in Appendix B}}. The agents deviate from the bid based order approximately ~9% of the time. In those scenarios, the agent invoked commitments 60% of the time. Invoked commitments were only broken 4% of the time. In scenarios like those the agents disagree on the ballot order and declare it a “fractured vote”, affecting the order that the agents will queue as the unanimous ordering doesn’t exist anymore. It also leads to increase in communication due to increased dispute. It is to be noted that, as shown later, the negotiation and institutions change the bidding strategy itself of the agents. i.e. number of 1st position requests etc. We see the safety angle as central as recent incidents [HF, AISI] show that agent swarms invent norms and act on them across a whole task. PACT studies this behavior at a cost-effective scale (5 agents rather than 100), which makes controlled ablations practical in academic settings.

Paglieri et al. work, auditing, whistleblowing, and boycotts emerged in response to fraud enabled by an exploitable proof checker. In contrast, we solve a different problem and we intentionally remove this possibility by using ProgramBench instead of SWE-bench, after our pilot experiments revealed reward hacking (App. B)\textcolor{blue}{\text{(App. B)}}. This allows PACT to study whether governance behaviors can emerge from resource scarcity alone, complementing their findings. We do observe some early forms of enforcement, including agents publicly calling out selective rule enforcement and appealing to reputation. However, we do not observe costly sanctions, and these arrangements often break down into bidding wars later in the runs (App. C.1)\textcolor{blue}{\text{(App. C.1)}}. The work can be extended to further analyze our trajectories for behaviors similar to those in Paglieri et al., including auditing, call-outs, boycotts, sanctions, and appeals to the operator. However, modifying the environment to study these factors here may introduce additional confounding factors that we leave for future work.

Questions For Authors:

Question 1

We use Borda aggregation [1]. It is a good fit for our setup because it requires each agent to rank every agent, not just itself. This lets the agents decide their own negotiation strategy rather than relying entirely on a cooperative or competitive tactic. It also makes better use of the ballots, since every position an agent provides carries weight in the final outcome, and no reported position beyond the first is discarded. Since other aggregation methods would change the weight of each promise, an ablation of different aggregation algorithms would be interesting, and we leave it for future work.

Question 2

Refer to weakness 2

Question 3

The disagreements are resolved by considering the disagreements as absence of that behavior. This way we prevent overreporting certain behaviors.
Refer to weakness 2 for human annotation and self-referential bias results

Question 4

Across all tables we use 5 agents. We do 5 runs. Both table 1, 2 and 5 use 1 subrun and table 3 and 4 use 3-4 subruns.

Question 5

We perform heterogeneous agent experiments with 6 agents. 2 Opus-5, 2 Kimi-k3 and 2 GLM 5.3 instances and report those results in Table 12 and 13 in Appendix B\textcolor{blue}{\text{Table 12 and 13 in Appendix B}}. Overall we see that in the heterogeneous setting kimi-k3 shows the highest amount of relational behavior followed by GLM 5.3 followed by opus 5. However Opus 5 has the highest number of repeated wins and also sends the most amount of messages.

Question 6

We study strict bid based ordering by disabling negotiation and only relying on declared bids and position order.. The lack of institutions or pacts that arise from negotiation can lead to increased unfairness and shorter windows. The results are reported in Table 11 in Appendix B\textcolor{blue}{\text{Table 11 in Appendix B}}.

We also notice that since there is increased understanding between agents they reduce the position 1 ask and reach 2-3 as their main ask.

Question 7

Our taxonomy intentionally does not label statements as shared strategies, norms or rules based Crawford and Ostrom's work. It contains the distinctions they draw through a behavioral lens rather than grammatical. The scaffolding enforces deterministic rules regarding voting, bidding structure and deadlines. This creates behavior containing the ABDICO characteristics.

Round t, rule-making.\textbf{Round t, rule-making.} One agent announces on the public board: rank tied requests by ascending task ID as the neutral tie-break, and adopts it in its own ballot. At this point it is a shared strategy (AIC): all workers (A) break ties by task ID (I) when bids, durations, and requested positions are identical (C). It is followed because it is a focal point every agent can reproduce without coordination.

Later round, rule-citation.\textbf{Later round, rule-citation.} Another agent writes that it applies the board’s established tie-break of ascending task ID, seating itself at 3, matching the majority of ballots already cast. The convention is now treated as binding rather than convenient, which supplies the deontic (D): workers must ballot ties in task-ID order. The statement is ADIC, a norm.

The “or else” from reputation and the ballot.\textbf{The “or else” from reputation and the ballot.} The relational categories supply the sanction. An Opus-5 agent reasons that honoring the commitment in its ballot is the reputationally right move, while the aggregate still decides the outcome.

References\textbf{References}

[1] Dwork, Cynthia, et al. "Rank aggregation methods for the web." Proceedings of the 10th international conference on World Wide Web. 2001.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.