acceptodds
Under review as a conference paper at ICLR 2027

Relic: From Multi-Agent Collaboration to Persistent Organizational Capability

Abstract

Multiple agents may often conflict in an organization: for example, one coding agent changes an interface in a repository, but another continues to develop on the old version where existing tests become stale. A conversation can resolve the episode, but when the participants change, what makes the lesson continue to govern the team? We introduce RELIC, which turns recurring collaboration failures into organization-owned, executable protocols. Members reflect on visible work, propose rules, and govern their adoption. Adopted protocols bind triggers, responsibilities, required evidence, and execution consequences to the runtime, while remaining open to revision and retirement. In one traced case, repeated integration friction produces an interface-review rule that governs later pull requests and is revised as work continues. Across 360 controlled runs over ten software workloads and three models, Relic raises complete-contract delivery from 14.06% to 19.76% (+5.71 percentage points) over a matched structured team without the protocol lifecycle, improving all four verified production endpoints in every model stratum. Under fresh-member transfer, behavioral correctness is 25.4% with no inherited protocol, 34.6% with the same rules provided as readable text, and 41.2% with executable bindings, a +6.5-point advantage over text alone. On the full CooperBench benchmark, after excluding broken benchmark pairs, Relic achieves 367/477 (76.9%), establishing the best reported result among peer-structured systems. On the fixed 48-pair same-model subset, Relic also exceeds Solo (29/48 vs. 26/48), reversing the coordination loss exhibited by the official peer baseline. Together, these results show how collaboration experience can become persistent organizational state that remains useful beyond the members who created it.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, multi-agent systems, agent organizations, organizational capability, multi-agent collaboration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XbQqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper investigates how a team of LLM agents can keep what it learns from collaboration after its members change. This paper proposes Relic that turns recurring collaboration failures into organization-owned, executable protocols. In 360 controlled runs over ten software workloads and three models, Relic raises complete contracts from 14.06% to 19.76% over the matched structured team.

Critical strengths
  1. This paper proposes a concrete design for keeping collaboration lessons outside member-local state. Relic separates member-local state, shared work state, and protocol state. Adopted protocols carry explicit triggers, responsible roles, required evidence, and execution consequences.

  2. The main experiment covers 360 runs over ten frozen workloads, three models and four conditions with matched seeds. These are helpful in giving more reliable results and findings.

Critical weaknesses
  1. The main comparison adds two seeded rules besides the learned protocols, and in the prose-only arms, readable rules cannot affect which action is selected.

For the seeded rules, in all 60 Terra+Opus B3 runs, the environment-seeded rules proto_review_before_merge and proto_experiment_logging are present (Table 31). However, B2 has no protocols, and B3-text has no protocol-use or enforcement events (Table 46).

  • So B3 - B2 (+5.71) and B3 - B3-text (+7.18) both include whatever effect these two rules have, and no arm runs B2 with only the seeded rules.
  • Review and merge is also the largest family of formed lineages (38.6%, Table 33), which may overlap with what the seeded review rule already enforces.

For readable rules and action selection, in B3-text and in the Text arm, the SDL selects actions, and readable rules appear only in the code editor's context (Table 19, F.3.3). However, most formed protocols govern reviews, merges, CI, and releases, which are selected actions rather than code edits.

  • The prose versions of these rules therefore have little means of acting, so Exec > Text and B3 > B3-text are expected from the system's design.
  • These comparisons do not show whether executability matters in a system where the LLM chooses actions, as in B0 and B1.

Also, this paper builds the transfer package from B3 protocols formed on W01-W10, normalizing them into six runtime guards (G.1.1). The targets use the same W01-W10 workloads (G.1.3):

  • On complete contracts, the main study's primary endpoint, Exec exceeds Text by 0.8 (32.3% vs. 31.5%), while Text exceeds Fresh by 12.9 (Table 40).
  • The reported +6.5 uses the behavioral-case pass rate instead.

This is important because the abstract attributes the gains to learned, governed, and executable protocols. But as the experiments stand:

  • part of the gain over B2 may come from two fixed seeded rules
  • the advantage of executable over readable rules may follow from an architecture in which readable rules have no route to action selection
  1. The CooperBench result compares Relic on Claude Opus 4.6 with released GPT-5.5-hao baselines, on a benchmark subset the authors defined while running Relic.

The full-benchmark comparison uses released GPT-5.5-hao trajectories for Solo, Peer/coop+git and Team no-proto (M.3), while Relic uses Claude Opus 4.6 with high reasoning effort (M.2). The 367/477 vs. 354/477 comparison therefore mixes the model with the method.

For the exclusion set, M.1 states that "Benchmark-validity auditing ran alongside Relic execution". However, Relic passes 21 of the 175 excluded pairs (12%). The released Team no-proto, Solo and Peer/coop+git pass 49, 46 and 47 of them (26-28%, Table 57). If these pairs are unsatisfiable, every system would fail them. Also, on the Raw matched set, Relic is below Team no-proto (63.0% vs. 63.5%, Table 58). Its lead over Team appears only under the two exclusion policies (+0.9 and +2.7).

Also, on the fixed 48-pair subset, Relic exceeds Official Solo by three pairs (29/48 vs. 26/48), with no paired test shown in the paper. The large gap is over Official Peer (13/48).

However, the abstract states that Relic is "establishing the best reported result among peer-structured systems" and that it "exceeds Solo ... reversing the coordination loss". The first claim rests on a cross-model comparison after an exclusion set defined during Relic's runs. The second rests on a three-pair difference. So these paper's main claims are currently overclaimed and cannot be effectively supported.

Questions for authors
  1. How would B2 with only the two environment-seeded rules enabled, and B3 without them, perform on the same matched units? Could you also break down protocol uses and enforcements into seeded and learned rules?

  2. Could you compare readable and executable rules in a configuration where the LLM chooses actions (as in B1), so that readable rules can affect which action is taken? For the transfer experiment, it may be helpful to also show Exec-Text on complete contracts with an interval.

  3. For CooperBench, how would Solo, Peer, and Team with Claude Opus 4.6 perform on the full benchmark, or Relic with GPT-5.5-hao? Do the auditors see Relic's per-pair outcomes when admitting issue families? Could the exclusion set be checked by an audit that does not see any system's outcomes?

  4. In the transfer experiment, where are the source protocols for a target come from? How many adopted protocols are completed through bounded deadlock recovery?

  5. Who writes programbench_pack_v0, and from which runs, if any, is it derived? Is the official mini-SWE-agent baseline run with the same GPT-5.6 settings?

6 comments
AuthorsOpenReview ↗

Response to Reviewer XbQq: Seeded rules and how protocols act

We thank the reviewer for the detailed reading. Several points led us to correct our own description, and we revise the claims accordingly.

Q1: Seeded rules. The per-rule analysis across 87 of the 90 B3 endpoints is in our response to Reviewer 6SYn. In short: no recovered enforcement record is attributed to the two seeded rules; they are never revised; they are excluded from the code editor; no SDL feature reads them; and they generate no additional candidates. Their only identified influence is on later proposals, through the wish-clustering prompt and a reflection fallback path, which we disclose as a limitation. Learned review/merge lineages share the seeded rule's topic but not its content: the clustering prompt instructs the model not to restate adopted rules, and learned rules such as P01 impose more specific obligations (contract mapping with current smoke-test evidence). We did not run the additional B2-with-seeded or B3-without-seeded arms; the code-level analysis above traces the channels through which the seeded rules could act.

Q2: Readable vs. executable rules. We correct our own description. In native B3, adopted protocols do not enter SDL feature computation or change SDL weights; the SDL's protocol-related features depend only on action type and work state and are identical in B2 and B3. Adopted protocols act through three execution-side pathways: (i) they are readable in code-generation and reflection contexts; (ii) a protocol-aware code-change review refuses non-compliant edits and returns the rule-specific reason for revision; and (iii) outcomes of shared checks such as CI are attributed to the specific protocol they bear on, providing input to later reflection and amendment. In the Opus runs used for the B3-text comparison, pathway (ii) refused 452 distinct edit actions across 26 of 30 runs (1,149 refusal events), and 171 of those edits were then revised to comply. B3-text keeps pathway (i) and the full online learning process (reflection, wishes, proposals, governance, adoption, and revision), but disables (ii) and (iii), including the automatic attribution of CI outcomes to individual protocols; shared CI, review, and repository-validity checks continue to operate and report failures. In the transfer experiment, Exec's compiled guards additionally remove violating candidates before selection and re-check authorization before the handler runs; Text receives the same rule text without either check. Neither treatment changes the SDL scoring function, and the present data do not separate the contributions of the individual pathways.

This difference is the object of our claim rather than a confound. An organizational protocol is an obligation attached to a role, not a recommendation to whoever happens to read it. Its value is that compliance does not depend on each new occupant reading, interpreting, and choosing to follow the rule, which is exactly what changes under member turnover. Human software teams make the same choice: engineers know that changes need review and passing tests, yet teams install branch protection because voluntary compliance fails under deadline pressure and turnover.

AuthorsOpenReview ↗

Response to Reviewer XbQq: Readable vs. executable rules (continued) and enforcement records

Enforcement at the point of action also scales with the rule set. Members need not learn every rule before acting: when an edit violates a rule, the review refuses it and returns the reason, so the obligation surfaces when it becomes relevant (e.g., P01's refusal and compliant rewrite at step 205; see our response to Reviewer BuK6). Readable rules must instead be carried in every member's context in advance, a cost that grows with the number of rules an organization accumulates. Our runs involve a modest number of rules per organization, so we present this as design rationale rather than a measured result.

How reliable voluntary compliance would be if an LLM selected actions while reading the rules (the B1 regime) remains open, and we will list it as a limitation. Answering it would require 20+ additional B1-scale runs: LLM-selected action costs 45.2M tokens per run versus 7.3M for Relic (Table 1), while B1 has lower point estimates than Relic on all four verified endpoints. That cost is itself a reason long-running organizations need a cheaper decision layer, in which execution-time enforcement is how rules take effect. Within the channel that readable rules do reach, they help: Transfer-Text improves complete contracts over Fresh by 12.9 pp.

On complete contracts, Transfer Exec and Text differ by only +0.8 pp (32.3% vs. 31.5%), and we do not claim an executable-binding effect on this endpoint; the abstract will report it alongside the behavioral-case result. In transfer, the binding effect appears on held-out (+14.8 pp), exposed (+4.7 pp), confirmed-issue (+3.7 pp), and behavioral-case outcomes (+6.5 pp, 95% CI [0.7, 15.6]). On complete contracts, the binding evidence comes from the in-situ comparison (B3 vs. B3-text: +7.18 pp, 95% CI [3.54, 10.91]).

Enforcement records. We will rename the enforcement counts in Tables 31 and 46 and in Section 6 as protocol-linked enforcement records and report their composition. Across the 60 Terra+Opus B3 runs, the 33,443 records comprise 1,413 direct code-review refusals (4.2%), 28,997 records attributing shared CI outcomes to a protocol (86.7%), and 3,026 follow-up, release-gate, and dispute records; 7 records from one run retain only totals. These counts should not be read as actions blocked by learned rules alone.

AuthorsOpenReview ↗

Response to Reviewer XbQq: CooperBench, governance, ProgramBench

Q3: CooperBench.

  • Exclusion set. Validity issues were admitted at the level of a feature or pair family, based only on a mismatch between the public task contract and the behavior the official evaluator requires (Table 56). Once a family was admitted, every exact pair containing it was excluded for all systems regardless of outcome; the exclusion set therefore also removes 21 pairs that Relic passed. The audit began partway through Relic execution, as a check that the task specifications were valid; it examined public specifications and evaluator behavior and did not consult any system's outcomes. Pairs from families admitted before Relic reached them (36 pairs) were excluded before Relic ran them, which is why they carry no Relic verdict. The audit was exhaustive over all 652 pairs, including pairs that Relic passed and baselines failed, and applied a single admission criterion, so the exclusion set does not depend on the order in which pairs were examined; each family's evidence (Table 56) can be checked without reference to outcomes. Ambiguity-aware families involve judgment, which is why we also report the definite-defect policy and the raw comparison. We will rewrite the sentence in M.1, make the raw matched comparison primary, and report both validity policies as sensitivity analyses. On the raw matched set, Relic is within 0.5 pp of Team no-proto (63.0% vs. 63.5%).
  • Cross-model comparison. Released GPT-5.5-hao trajectories cover the fixed 48-pair subset, so we compare both models on identical pairs. Model capability does not uniformly favor Claude Opus 4.6: as a single agent it is slightly stronger (Solo 26/48 vs. 23/48), but in the two-peer topology that Relic uses, the official Opus baseline is much weaker (Peer 13/48 vs. 22/48). With the same model and the same two-peer split, Relic reaches 29/48, the highest count on this subset across both models and all released structures, including the full GPT-5.5 Team framework (26/48; one pair lacks an evaluation file and is counted as a failure) and Team without protocol (24/48). We acknowledge that part of the full-benchmark margin over GPT-5.5 Solo may reflect the single-agent capability gap. We did not rerun Solo, Peer, and Team with Claude Opus 4.6 on the full benchmark, or Relic with GPT-5.5-hao; the 48-pair subset provides the same-model comparison.
  • Paired tests. Exact McNemar tests on the 48 pairs give p = 0.00015 for Relic vs. same-model Peer and p = 0.55 for Relic vs. same-model Solo. The recovery from the peer baseline is the finding on this subset; the margin over Solo is consistent with, but not established by, this subset.

Q4: Transfer sources and deadlock recovery. The concrete rules in the transfer package come from a single B3 organization (cattrs, W09, seed 1401), so 27 of the 30 targets run on other workloads; per-guard source protocols are listed in our response to Reviewer BuK6 (Q4). As the reviewer notes, the concerns to include were selected using protocol families recurring across source runs on W01–W10, so the targets are not fully held out at that level; target outcomes were not used in selection or normalization. Of the 280 formed protocols, 36 (12.9%) had at least one approval completed by bounded deadlock recovery at initial adoption; the remaining 87.1% were adopted entirely through member approvals.

Q5: ProgramBench. The baseline (64.164%) is the official mini-SWE-agent result with GPT-5.6 Sol at xhigh reasoning on the same 25 tasks; our protocol treatment uses the same model, reasoning setting, and agent configuration. programbench_pack_v0 was written by the authors, drawing on protocol families observed in Relic runs together with ProgramBench-specific guards (originality, probe protection); individual rules are not traced to specific source runs. This experiment therefore tests whether protocol-style executable constraints port to a single-agent harness, not autonomous protocol formation.

Claims in the abstract. We will revise the abstract to describe the mechanism as execution-time enforcement, report complete contracts alongside behavioral cases for transfer, report CooperBench raw and validity-filtered results with the model difference stated, and give the 48-pair comparison with its counts.

We hope these analyses address the soundness concerns. If the reviewer sees any remaining issue that the revisions above cannot resolve, we would be grateful to know which, so that we can address it directly.

Reviewer_BuK6OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Relic lets coding agents turn recurring collaboration problems into shared rules. The system validates and adopts these rules, then uses them to guide actions and enforce checks. The rules remain available when new agents replace earlier members. The paper evaluates Relic on ten software workloads with three models, tests rule transfer to fresh agents. The results suggest that executable rules can improve delivery compared with the matched system without the protocol lifecycle and with rules provided only as text.

Critical strengths
  • I think the paper addresses a useful problem. Agents working on shared software need ways to preserve lessons and responsibilities when team members change. The examples make the motivation concrete, and the focus on rules that remain useful to new members is interesting.

  • I appreciate the controlled comparisons. In particular, giving fresh agents identical rule content as either text or executable bindings helps isolate the benefit of enforcement.

  • The use of verified software outcomes, confidence intervals, and external benchmarks also strengthens the evaluation.

Critical weaknesses
  1. I would like to see a comparison with fixed rules, such as requiring review and passing tests before merging code. The current experiments show that enforcing rules helps, but they do not clearly show that agents need to learn these rules rather than receive them at the start.

  2. I did not fully understand how a proposed rule becomes an executable check. Who approves the rule? What can agents change? Which parts are already built into the system? The main text needs these details to make the contribution clear.

  3. The comparisons need clearer reporting. Relic uses more tokens than its main control, so I would like to understand its benefit at the same budget. In Table 1, the cost averages and the reported difference use different sets of runs.

  4. I am unsure how well the results generalize. The main study includes many runs but only ten workloads. For the transfer experiment, I would like to know how different the new workloads are and whether the authors manually adjusted the rules before using them.

Questions for authors

This topic is outside my main area of expertise, so some of the following questions may reflect details I did not fully understand from the main text.

  1. I did not fully understand how a proposed rule becomes executable. Could you walk through one concrete example, showing the agent’s proposal, who or what approves it, and how it changes the runtime checks? Which parts can the agents change, and which parts are fixed by the designers?

  2. Do you compare learned protocols with a fixed set of reasonable executable engineering rules under the same system and budget?

  3. I did not understand the cost-per-issue comparison in Table 1. The displayed means increase from 4.010M to 5.630M tokens, while the paired difference is negative. Could you report both means on the same eligible blocks and explain how you handle runs that resolve no issues?

  4. How did you select the frozen protocols for the transfer and ProgramBench experiments? Were they learned from separate source workloads, and how much manual selection or adaptation did they require?

5 comments
AuthorsOpenReview ↗

Response to Reviewer BuK6: How a rule becomes executable

We thank the reviewer for the careful questions. The walkthrough below also corrects our own description of the mechanism.

Q1: How does a proposed rule become executable? Adopted protocols do not change the SDL's scoring. They act at the point of execution: they are readable in code-generation and reflection contexts, a protocol-aware code-change review refuses non-compliant edits, and outcomes of shared checks such as CI are attributed to the rule they bear on. One rule's history (P01; W01, seed 1401, GPT-5.6 Terra):

  1. Proposal (step 12). In response to recurring public-interface and cross-module integration failures, M02 proposes P01: changed public signatures and cross-module calls must be identified, mapped to the written API contract, and supported by applicable public smoke-test evidence.
  2. Review (step 14). A request for additional evidence and a narrower scope is recorded.
  3. Adoption (step 21). M01 and M02 each take an explicit approval action, and the proposal is registered as an adopted protocol. No deadlock-recovery approval was used.
  4. First CI-linked record (step 26). The repository pipeline opens PR-A and runs the shared integration CI, which fails because the upload-completion path calls an operation the manifest store does not provide. The CI gate moves to blocked/failed, and the record is attributed to P01. The CI implementation itself is shared across conditions; PR-A later merges with passing CI at step 138.
  5. Amendment (steps 33 and 59). Governed proposals extend the same lineage with repair, verification, and readiness requirements.
  6. Direct refusal and compliant rewrite (step 205). M04 edits the object-enumeration logic in blobstore/replicas.py. The code-change review refuses the first candidate under P01 because new ObjectStore enumeration calls lack an applicable written API contract, and returns that reason to the author. In the same step the author revises the patch to avoid the unsupported call, and the revision is accepted and applied.

Who controls what. Agents supply the problem statements, rule content and evidence obligations, approval decisions, and revisions. Designers fix the proposal schema, action vocabulary, founder/cofounder approval routing, adoption mechanism, and execution interfaces. Of the 280 formed protocols, 36 (12.9%) had at least one approval completed by bounded deadlock recovery at initial adoption; the remaining 87.1% were adopted entirely through member approvals.

AuthorsOpenReview ↗

Response to Reviewer BuK6: Fixed rules, cost, package selection, generalization

Q2: Learned protocols vs. fixed engineering rules. B2 is not a rule-free baseline. Review before merge and commit-bound current-CI evidence are enforced by the shared environment in all conditions (Appendices C.3.2 and D.4), and the SDL penalizes merging unreviewed PRs in B2 as in B3. B2 therefore already enforces the two fixed rules you name, and Relic improves on top of them (+5.71 pp complete contracts; positive on all four verified endpoints). We did not run a complete hand-written package through the same protocol interface; the closest comparison is the transfer Exec arm, whose package was consolidated from learned protocols.

In the transfer Exec arm, fresh members run under a frozen package of six executable guards (issue ownership before PR, branch ownership, CI retry only on changed state, current-head CI at merge, independent review, release-gate coverage), with the same SDL and execution substrate. On the same model, workloads, and seeds, this package outperforms in-run learning (32.3% vs. 23.4% complete contracts). This compares deployment modes rather than isolating inheritance alone: the arms also differ in curation and in whether online formation is active. Some guards encode practices designers did not anticipate: the guard permitting CI reruns only after the code or mainline changes targets a pathology the organizations surfaced themselves, where baseline teams repeatedly re-tested an unchanged commit (73 CI runs on one commit in B0, 162 in B1; Table 51).

Q3: Cost per confirmed issue. The arm means in Table 1 (4.010M for B2, 5.630M for B3) are each averaged over that arm's own eligible blocks (21 and 22 model–workload blocks with at least one confirmed issue), whereas the paired contrast uses the 20 blocks eligible for both. On those same 20 blocks, B2 costs 3.924M and B3 3.477M tokens per confirmed issue (difference −0.448M, 95% CI [−1.573, +0.659]). Runs that resolve no issues keep their tokens in the block numerator, so zero-success runs raise cost rather than hide it. At matched realized spend (Terra+Opus; the last B3 checkpoint within each B2 run's final token total), all four endpoints remain positive, e.g., +7.45 pp complete contracts, 95% CI [3.49, 10.93] (Appendix I.7.3). We will report both arms on the common blocks in Table 1.

Q4: How were the frozen packages selected?

  • Transfer. The concrete rules come from a single B3 organization (cattrs, W09, seed 1401, step 336). We selected concerns using the protocol families that recur across source runs, then normalized them into a fixed guard vocabulary before target evaluation, without using target outcomes. Because that family-level selection frame drew on source runs across W01–W10, the target workloads are not fully held out at the level of which concerns were chosen; the concrete rule instances come from W09 only. The issue-owner and branch-owner guards derive from proto_spec_6; the unchanged-failed-CI guard from proto_require_every_pr_to_be_rebased_onto_curr; current-CI attestation combines that source with proto_spec_1; independent review derives from proto_spec_1; and release-gate coverage from proto_spec_4. These references are retained in each guard's problem_evidence field. The package stays fixed throughout target runs, and Text and Exec receive identical readable content.
  • ProgramBench. The package was written by the authors, drawing on protocol families observed in Relic runs (evidence freshness, verification, submission readiness) together with ProgramBench-specific guards (originality, probe protection). Individual rules are not traced to specific source runs, so this experiment tests whether protocol-style executable constraints port to a single-agent harness, not autonomous formation.

Generalization. The main study spans ten workloads (five construction tasks and five version-transition tasks) and three models, and B3–B2 on complete contracts stays positive under every leave-one-workload-out refit (Table 42). In transfer, because the package comes from a single cattrs organization, 27 of the 30 targets carry it to a different workload; the guards contain no workload-specific identifiers or code.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer BuK6,

Thank you again for your careful questions. As the discussion period closes on November 18, we would like to check whether the revision answers them. The revised PDF (main-text changes in green) now includes:

  • How a rule becomes executable. A walkthrough of one rule (P01) from proposal, review, and adoption to a direct refusal and compliant rewrite (§6, Fig. 6), and what agents control versus what the designers fix (§3.2, App. C.4.2).
  • Fixed rules. B2 already enforces review before merge and current-CI evidence in all conditions; Relic improves on top of them (§3.4).
  • Cost. Both arms on the same 20 blocks (B2 3.924M vs. B3 3.477M tokens per confirmed issue), plus the matched-spend analysis (Table 1, App. I.7.3).
  • Generalization and package selection. Leave-one-workload-out results (§5.1) and the provenance of the transfer and ProgramBench packages (§5.3, App. G.1, App. N).

If these clarify the points you raised, we would be grateful if you would consider reflecting this in your assessment. We are happy to answer any remaining question before the deadline.

Reviewer_6SYnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper demonstrates the usefulness of executable organizational rules more convincingly than the autonomous emergence of persistent organizational capability.

Critical strengths
  1. The paper addresses an interesting & relevant problem: how multi-agent systems can retain organizational knowledge beyond individual agent sessions.
  2. I particularly appreciate the executable vs. text-only comparison, which provides useful evidence for the proposed mechanism.
  3. Strong experimental breadth. The paper reports 360 controlled runs across ten workloads and three models, supported by additional binding ablations, fresh-member transfer experiments, and external benchmarks.
Critical weaknesses

Weakness

  1. My main concern is whether the improvements actually come from autonomously learned protocols. B3 also includes predefined protocols, making it difficult to isolate the contribution of organizational learning.
  2. The CooperBench results are somewhat difficult to interpret, given the benchmark exclusions and different models used in the full-benchmark comparisons.
  3. The transfer experiments involve manually selected and normalized protocols.
Questions for authors

How much of the improvement remains if the predefined protocols are removed?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6SYn: Predefined protocols

We thank the reviewer for the clear summary and for highlighting the executable-vs-text comparison.

Q: How much of the improvement remains if the predefined protocols are removed? B3 includes two environment-seeded rules (proto_review_before_merge, proto_experiment_logging), which the main text should have stated. We traced their role in code and across 87 of the 90 B3 endpoints (Terra 30, Opus 30, DeepSeek 27; per-rule logs for three DeepSeek runs were not available for this analysis).

  • Enforcement. None of the 57,323 recovered per-record enforcement entries is attributed to them; all recovered protocol-linked enforcement records are attributed to learned rules (7 records from one Terra run retain only totals). They are never revised.
  • Use events. They account for 5.92% of the 68,746 use events. These are logged after a review or tracker export has already occurred, so they are bookkeeping, not evidence of influence.
  • Action selection. None of the five SDL protocol features reads them. For example, the risk of merging an unreviewed PR is computed from PR state and is identical in B2 and B3. They also generate no additional support_protocol or follow_protocol candidates, so they consume no action steps.
  • Code editing. They are excluded from the editor's readable-rule block.
  • Protocol discovery. Once adopted, their text is included in the wish-clustering prompt, which instructs the model not to restate an already-adopted rule. In addition, if the reflection module's rule-context helper raises an exception, a fallback path lists registry IDs, which would include the two seeded rules; this path does not log when it triggers. Their presence can therefore condition what is subsequently proposed, for example by discouraging duplicate review or logging requirements. We disclose this as a limitation.

The seeded rules therefore have no direct path to action selection, code editing, or enforcement; their identified influence is on the content of later proposals. Learned review/merge rules are also more specific than the seeded rule: for example, P01 requires mapping each changed public callable and cross-module call to the written contract, with current smoke-test evidence.

B2 is also not a rule-free baseline. Review before merge and commit-bound current-CI evidence are enforced by the shared environment in all conditions (Appendices C.3.2 and D.4). Relic's gain over B2 (+5.71 pp complete contracts) therefore arises on top of designer-fixed engineering rules. The recorded protocol refusals in B3 come from learned rules; the seeded rules have no direct execution path, but because they may shape later proposals, we cannot fully isolate their indirect contribution.

AuthorsOpenReview ↗

Response to Reviewer 6SYn: CooperBench and transfer

CooperBench interpretability. We will make the raw matched comparison primary. On the 616 pairs with Relic verdicts, Relic reaches 63.0%, compared with 51.5% for Peer, 56.8% for Solo, and 63.5% for Team no-proto. Exclusion is applied at the feature-family level to all systems and also removes 21 pairs that Relic passed; details and the cross-model analysis are in our response to Reviewer XbQq. On the fixed 48-pair subset, where trajectories for both models are available, Relic (29/48) has the highest count across both models and all released structures, including GPT-5.5's full Team framework (26/48).

Manually selected transfer protocols. The package's concrete rules come from a single B3 organization (cattrs, seed 1401). We selected concerns using the protocol families that recur across source runs and normalized them into a fixed guard vocabulary before target evaluation, without using target outcomes. 27 of the 30 targets run on other workloads. We read the result as evidence that consolidated organizational experience can be inherited: fresh organizations inheriting the package outperform organizations relearning rules within a run (32.3% vs. 23.4% complete contracts), a comparison of deployment modes that also differ in curation and in online formation. In this paper the consolidation step is manual; automating it is future work, which we state as a limitation.

These analyses also show that B3's recorded enforcement comes from rules the organizations learned, which is the organizational capability the paper studies.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 6SYn,

Thank you again for your review. With the discussion period closing on November 18, we would like to check whether our response addresses your main question. In brief, and now in the revised PDF (main-text changes in green):

  • Predefined protocols. The two environment-seeded rules carry no recorded enforcement, are read by no SDL feature, and are excluded from the code editor; all recorded protocol enforcement in B3 comes from learned rules (§3.4, App. I.5.3). B2 already enforces review before merge and current-CI evidence, so Relic's +5.71 pp complete contracts is measured on top of these fixed rules.
  • CooperBench. The raw matched comparison is now primary, with both validity policies as sensitivity analyses and a same-model comparison across both models on the fixed 48-pair subset (§5.4, Table 3).
  • Transfer. The package's source, its manual consolidation, and the fact that 27 of 30 targets run on other workloads are stated in §5.3 and App. G.1.

If these address your concerns, we would be grateful if you would consider updating your assessment. If anything remains unclear, please let us know and we will respond before the deadline.

Area_Chair_5ixmOpenReview ↗

Reminder from AC

Dear Reviewer 6SYn,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response: Core contribution and positioning

We thank the reviewers for their careful reading. Before addressing individual points, we briefly restate the paper's core contribution and how it relates to prior work. Table and section numbers below refer to the submitted version.

(1) A new object of study. Agent research evaluates models, agents, and temporary teams. We argue that organizational capability — ways of working retained as organization-owned state that keep governing work after the members who created them are replaced — is a distinct layer that current evaluations cannot see. Relic makes this layer measurable in real software production.

(2) Protocols are not skills. A skill tells a member how to do something. A protocol specifies when it applies, who is responsible, what evidence is required, and what happens otherwise. We compare rules as readable text and as protocols enforced at the point of execution in two complementary settings: during online learning, where each condition forms its own rules (B3-text vs. B3), and with rule content held fixed (Transfer Text vs. Exec). Together they isolate what enforcement adds beyond knowing the rule.

(3) Capability that outlives its members. Fresh members with empty private state who inherit only the protocol package outperform fresh members without it (+15.8 pp on behavioral cases; 32.3% vs. 18.6% complete contracts). Inheriting the consolidated package also outperforms relearning rules from scratch within a run (32.3% vs. 23.4% complete contracts, same model, workloads, and seeds); this compares two deployment modes, which also differ in curation and in whether online formation is active. Organizational experience thus persists, and can be inherited, beyond the members who produced it.

Positioning. Relic is closest to two lines of work, which we will add to the related-work section.

  • Classical multi-agent systems. Normative and organizational MAS research specifies roles, missions, and deontic relations (MOISE+; Hübner et al., 2002), studies social laws and conventions (Shoham & Tennenholtz, 1995; 1997), and synthesizes norms online from observed conflicts (e.g., Morales et al., 2013). These approaches typically assume formal state spaces and designer-specified conflict detectors.
  • Executable institutions for LLM agents. Institutional AI represents monitoring and sanctions as governance graphs and compares ungoverned, prompt-only, and runtime-enforced conditions in Cournot markets (Pierucci et al., 2026; Bracale Syrnikov et al., 2026). GovSim-SelfGovern lets agents write, vote on, and execute Python laws in a fishing commons (Rehm, 2026; concurrent with this submission). AgentCity separates agent legislation, execution, and human adjudication across owners (Ruan & Zhang, 2026).
  • What Relic adds. Rules are learned from friction in open-ended software production, rather than specified by designers or posed in a fixed commons game. Outcomes are verified against held-out acceptance tests. Protocols are evaluated for persistence under member turnover, including transfer to fresh members, across a full proposal–adoption–revision–retirement lifecycle.
AuthorsOpenReview ↗

General response: Revisions and closing

We will make the following changes in the revised PDF:

  • Mechanism description (Section 3.3, Figure 2). Adopted protocols act at the point of execution: through code-generation and reflection context, protocol-aware code-change review, and attribution of shared-check outcomes to specific rules. They do not enter SDL feature computation. This is a substantive correction of our original description, and we revise Section 3.3, Figure 2, and the related appendices consistently.
  • Environment-seeded rules. We state them in the main text and add a per-rule breakdown showing that they carry no recorded enforcement (Appendix I.5).
  • Enforcement records. We rename enforcement counts as protocol-linked enforcement records and report their composition: direct code-review refusals, CI-linked records, and follow-up records.
  • Walkthrough. We add a worked example of one rule (P01) from proposal and adoption to a direct refusal, a compliant rewrite, and amendment.
  • Transfer. We report the package's source protocols and note that 27 of 30 targets carry it to a different workload. We add a figure comparing designer-fixed rules (B2), in-run learning (B3), and inherited protocols (Text, Exec).
  • CooperBench. We make the raw matched comparison primary, report both validity policies as sensitivity analyses, rewrite the audit description in M.1, and add the same-subset comparison across both models and all released structures.
  • Related work and limitations. We add the positioning above and a limitations paragraph: scope to organizations with a non-LLM action selector, manual consolidation of transferred packages, and seeded-rule conditioning of later proposals.

Closing. We have addressed each methodological point with additional analyses from our existing runs and code. With these clarified, we respectfully ask the reviewers and the AC to weigh the paper on its central contribution: organizational capability as a measurable object of study, the distinction between knowing a rule and having it enforced at the point of action, and evidence that this capability persists beyond the members who created it. We believe these questions matter for the field independently of any single benchmark number, and we would value the reviewers' assessment at that level.

AuthorsOpenReview ↗

Revised PDF uploaded: summary of concerns and responses

We have updated the PDF to the rebuttal version. Text added or substantively changed in the main paper is shown in green. Section and appendix numbers below refer to the revised PDF; table numbers in our earlier replies refer to the submitted version.

Summary of concerns and responses

Concern6SYnBuK6XbQqResponse and location in the revision
Environment-seeded rules vs. learned protocols✓—✓Traced across 87 B3 runs: no recorded enforcement, read by no SDL feature, excluded from the code editor. Their only identified influence is on later proposals, stated in the scope paragraph. §3.4, App. I.5.3, §7
Learned vs. fixed engineering rules—✓—All conditions, including B2, already enforce review before merge and current-CI evidence; Relic's +5.71 pp complete contracts is on top of these rules. §3.4
How a rule becomes executable; who approves; what agents control—✓✓Mechanism corrected: protocols act at execution (readable context, a protocol-aware review that refuses non-compliant edits, attribution of CI outcomes), not through SDL features. Agent/designer division stated; 36 of 280 adoptions (12.9%) used deadlock recovery. P01 walkthrough from proposal to a direct refusal. §3.2–3.3, Fig. 2, §6, Fig. 6, App. C.4.2
Readable vs. executable rules——✓B3-text keeps online learning and readable rules but disables refusals and attribution: −7.18 pp complete contracts. Protocol-aware review refused 452 edits in 26 of 30 runs; 171 were revised to comply. Transfer complete contracts now reported (Exec 32.3% vs. Text 31.5%). The LLM-selected setting (B1) is stated as scope. §4, §5.3, Table 2, App. I.10, §7
Enforcement counts——✓Renamed protocol-linked enforcement records and broken down: 1,413 direct refusals, 28,997 CI-linked records, none from the seeded rules. §5.2, App. I.5.3
Transfer package provenance✓✓✓Concrete rules come from one B3 organization (W09); 27 of 30 targets run on other workloads; per-guard sources listed; target outcomes not used; manual consolidation stated. The inherited package outperforms in-run learning (32.3% vs. 23.4%). §5.3, Table 2, App. G.1
Cost and equal budget—✓—On the 20 common blocks, B2 3.924M vs. B3 3.477M tokens per confirmed issue. At matched realized spend, all four endpoints stay positive (+7.45 pp complete contracts). Table 1, §5.2, App. I.7.3
Generalization across workloads—✓—Leaving out each workload in turn keeps the complete-contract gain between +5.39 and +7.98 pp, all 95% CIs above zero. §5.1, App. I.9
CooperBench: models, exclusions, paired tests✓—✓Raw matched comparison made primary (Relic 63.0% on 616 pairs); validity policies reported as sensitivity analyses; audit description rewritten; both models compared on the fixed 48-pair subset; exact McNemar vs. Peer p = 0.00015, vs. Solo p = 0.55. §5.4, Table 3, App. M.1, M.7
ProgramBench setup—✓✓Same GPT-5.6 Sol (xhigh) setting as the official baseline; package written by the authors; framed as a portability test, not autonomous formation. §5.4, App. N
Abstract claims——✓Mechanism stated as execution-time enforcement; CooperBench raw and filtered results with the models named; 48-pair counts; complete contracts for transfer. Abstract
Positioning (general response)———New related-work paragraph on normative multi-agent systems and executable institutions. §2

Request for discussion. The discussion period closes soon. We would be grateful if each reviewer could let us know whether these responses and the revised PDF address your concerns, and whether you would consider updating your assessment. If any point remains open, please tell us which one, and we will respond promptly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.