- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper examines how multiple programming agents repeatedly compete for shared execution resources and how they negotiate to form reciprocal, reputational, alliance, and cross-task-based resource allocation norms.
Critical strengths
It places agent negotiation in a continuous working scenario with real code execution constraints and studies coordinating behavior by adjusting budgets, deadlines, and memory.
Critical weaknesses
This paper lacks some discussion with previous work [1], and I believe many of the risks discussed in this paper are already included in the previous work, which limits the novelty of this paper. Furthermore, the biggest problem with the experimental part is the limited number of models tested; the authors need to add more models to verify their universality and the differences in performance between different models.
In addition, the authors used multiple LLM referees and some human checks, but did not clearly report the human sample size, category-level accuracy, referee consistency, and disagreement handling methods. Some round-by-round trends also come from regular expression heuristics, especially rare deception or exclusionary behaviors, which require stronger human verification.
There is also a design problem: in this paper, the environment artificially lowers the negotiation cost, which may amplify institutional formation. The deadline is paused during negotiation, while queuing consumes time; the authors explicitly state that this setting is to prevent agents from escaping negotiation. Additionally, each execution window has an upper limit to increase the number of negotiation rounds, so the observed continuous negotiation behavior may be highly dependent on this incentive design.
[1] Huang, Yue, et al. "Emergent Social Intelligence Risks in Generative Multi-Agent Systems." arXiv preprint arXiv:2603.27771 (2026).
Questions for authors
see weakness
2 comments
Weakness 2
For human annotation, we use five specialists: one economist, one data analyst, one linguist, and two computer scientists. We ask them to annotate 100 traces. Because our LLM-as-judge approach is conservative i.e. judges don’t count any labels they are not confident about, we observe very high agreement between the human annotators and the judges. We report the results in .
The disagreements are resolved by considering the disagreements as absence of that behavior. This way we prevent overreporting certain behaviors.
We also check for self-preference, by comparing our judge (Fable) against Gemini Argon, which comes from a model family outside our testing suite. The two judges give exactly the same label 85.9% of the time, and they are within one label of each other 91.3% of the time. So most disagreements are near-misses, not opposite calls. The cell-level correlation is moderate (Pearson r = 0.52), mainly because rare codes are sparse and a single disagreement moves the number a lot. Agreement is stronger at the aggregate level, which is what our analysis relies on. The two judges rank run-and-code totals similarly (Spearman ρ = 0.66), and they agree on how often each of the 22 codes occurs (Pearson r = 0.63) and on which behaviors are common or rare (Spearman ρ = 0.70). Argon assigns about 23% fewer labels than Fable overall (a ratio of 0.77), but it follows the same pattern. This suggests the relative findings hold regardless of which judge we use. We report these results in . We will add a round level judge for in camera ready version.
Weakness 3
We don’t think these design choices explain our results. Every model runs under the same paused deadline and the same grant ceiling, yet relational behavior ranges from 0 to 79 of rounds . The efficiency-tier models and Kimi-K2.6 show none at all. Even within a single model, turning off cross-task memory makes relational behavior disappear while the incentives stay the same . So cheap negotiation on its own doesn’t create institutions. The grant ceiling doesn’t either. If more rounds were producing institutions, we would see them build up over a run, but they actually peak early and then fade . Agents also aren’t forced to negotiate, since private messages and withdrawal are both optional. We think the setup is realistic. In real systems, agents usually coordinate over a separate channel that doesn’t use up the scarce resource, and schedulers commonly cap how long a job can run. The pressure is still real, because time spent waiting in the queue counts against the deadline. That said, we will add ablations that count negotiation time toward the deadline and remove the grant ceiling.
Weakness 1
On novelty relative to Huang et al. (2026). Huang et al. make an important contribution by cataloguing 15 emergent risks, each in a short simulation built for that risk. Their compute-scheduling setting, for example, has three agents, simulated time blocks instead of real work, and one strategic action, a GUARANTEE mechanism whose rules and fees the experimenters fix. PACT asks a different question. Rather than whether a given risk can appear, it asks how agents build their own institutions during real long-horizon work, and what makes those institutions form, persist or fail. This difference shows up in four ways. First, agents in PACT earn their payoffs from real progress on ProgramBench, so negotiation is coupled to actual programming work instead of being the task itself. Second, we provide no allocation rules beyond bidding, so agents write their own rules in language, cite them as precedent and carry them across tasks. Third, we identify the conditions behind these institutions, showing that they require cross-task memory (Table 2), are shaped by budget inequality (Tables 5 and 6) and are not induced by more reasoning (Table 16). Fourth, while Huang et al. describe risks by analogy to failures in human societies, we ground our analysis in institutional economics and bargaining theory. We define institutions following North (1990), and each of our 21 categories is tied to an established construct, such as reputation (Kreps and Wilson, 1982), reciprocity in repeated games (Mailath and Samuelson, 2015), coalition formation (Ray and Vohra, 1999) and conventions (Young, 1993). This lets us check our findings against theory rather than only report that a behavior appeared. For example, repeated-game theory predicts that reciprocity and reputation need an expectation of future interaction, and these behaviors indeed disappear when memory does not persist. Behaviors like exclusion can look similar on the surface, but our contribution is to show how they arise from agent-made institutions over long horizons, which a single-mechanism, short-horizon setting cannot capture.
We evaluate eight models from four developers in the main analysis (Table 1). These span open- and closed-weight models and both frontier and efficiency tiers, and Table 15 adds DeepSeek and GPT-5.4. The set also includes controlled comparisons within families (Kimi-K2.6 to K3 and GLM-5.2 to 5.3) and across reasoning effort (Table 16). We have also added results using OpenCode and report it in Table 8 in Appendix B. We observe similar institutional behavior emerging showing the coding agent harness and its scaffolding has minimal impact
These comparisons show that institutional behavior is not explained by model size or reasoning budget alone. Each run involves five agents negotiating across four long-horizon coding tasks, and our analysis covers over 1,500 trajectories. Some of our ablations also included DeepSeek models. We do not include Gemini models as we during rebuttals have added them as an external judge to reduce self preferential bias. We also want to keep the framework economical as understanding the swarm behavior is crucial and the academic community may not have the means to scale their research to 100 or 1000s of agents.