- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper investigates how a team of LLM agents can keep what it learns from collaboration after its members change. This paper proposes Relic that turns recurring collaboration failures into organization-owned, executable protocols. In 360 controlled runs over ten software workloads and three models, Relic raises complete contracts from 14.06% to 19.76% over the matched structured team.
Critical strengths
-
This paper proposes a concrete design for keeping collaboration lessons outside member-local state. Relic separates member-local state, shared work state, and protocol state. Adopted protocols carry explicit triggers, responsible roles, required evidence, and execution consequences.
-
The main experiment covers 360 runs over ten frozen workloads, three models and four conditions with matched seeds. These are helpful in giving more reliable results and findings.
Critical weaknesses
- The main comparison adds two seeded rules besides the learned protocols, and in the prose-only arms, readable rules cannot affect which action is selected.
For the seeded rules, in all 60 Terra+Opus B3 runs, the environment-seeded rules proto_review_before_merge and proto_experiment_logging are present (Table 31). However, B2 has no protocols, and B3-text has no protocol-use or enforcement events (Table 46).
- So B3 - B2 (+5.71) and B3 - B3-text (+7.18) both include whatever effect these two rules have, and no arm runs B2 with only the seeded rules.
- Review and merge is also the largest family of formed lineages (38.6%, Table 33), which may overlap with what the seeded review rule already enforces.
For readable rules and action selection, in B3-text and in the Text arm, the SDL selects actions, and readable rules appear only in the code editor's context (Table 19, F.3.3). However, most formed protocols govern reviews, merges, CI, and releases, which are selected actions rather than code edits.
- The prose versions of these rules therefore have little means of acting, so Exec > Text and B3 > B3-text are expected from the system's design.
- These comparisons do not show whether executability matters in a system where the LLM chooses actions, as in B0 and B1.
Also, this paper builds the transfer package from B3 protocols formed on W01-W10, normalizing them into six runtime guards (G.1.1). The targets use the same W01-W10 workloads (G.1.3):
- On complete contracts, the main study's primary endpoint, Exec exceeds Text by 0.8 (32.3% vs. 31.5%), while Text exceeds Fresh by 12.9 (Table 40).
- The reported +6.5 uses the behavioral-case pass rate instead.
This is important because the abstract attributes the gains to learned, governed, and executable protocols. But as the experiments stand:
- part of the gain over B2 may come from two fixed seeded rules
- the advantage of executable over readable rules may follow from an architecture in which readable rules have no route to action selection
- The CooperBench result compares Relic on Claude Opus 4.6 with released GPT-5.5-hao baselines, on a benchmark subset the authors defined while running Relic.
The full-benchmark comparison uses released GPT-5.5-hao trajectories for Solo, Peer/coop+git and Team no-proto (M.3), while Relic uses Claude Opus 4.6 with high reasoning effort (M.2). The 367/477 vs. 354/477 comparison therefore mixes the model with the method.
For the exclusion set, M.1 states that "Benchmark-validity auditing ran alongside Relic execution". However, Relic passes 21 of the 175 excluded pairs (12%). The released Team no-proto, Solo and Peer/coop+git pass 49, 46 and 47 of them (26-28%, Table 57). If these pairs are unsatisfiable, every system would fail them. Also, on the Raw matched set, Relic is below Team no-proto (63.0% vs. 63.5%, Table 58). Its lead over Team appears only under the two exclusion policies (+0.9 and +2.7).
Also, on the fixed 48-pair subset, Relic exceeds Official Solo by three pairs (29/48 vs. 26/48), with no paired test shown in the paper. The large gap is over Official Peer (13/48).
However, the abstract states that Relic is "establishing the best reported result among peer-structured systems" and that it "exceeds Solo ... reversing the coordination loss". The first claim rests on a cross-model comparison after an exclusion set defined during Relic's runs. The second rests on a three-pair difference. So these paper's main claims are currently overclaimed and cannot be effectively supported.
Questions for authors
-
How would B2 with only the two environment-seeded rules enabled, and B3 without them, perform on the same matched units? Could you also break down protocol uses and enforcements into seeded and learned rules?
-
Could you compare readable and executable rules in a configuration where the LLM chooses actions (as in B1), so that readable rules can affect which action is taken? For the transfer experiment, it may be helpful to also show Exec-Text on complete contracts with an interval.
-
For CooperBench, how would Solo, Peer, and Team with Claude Opus 4.6 perform on the full benchmark, or Relic with GPT-5.5-hao? Do the auditors see Relic's per-pair outcomes when admitting issue families? Could the exclusion set be checked by an audit that does not see any system's outcomes?
-
In the transfer experiment, where are the source protocols for a target come from? How many adopted protocols are completed through bounded deadlock recovery?
-
Who writes
programbench_pack_v0, and from which runs, if any, is it derived? Is the official mini-SWE-agent baseline run with the same GPT-5.6 settings?
6 comments
Response to Reviewer XbQq: Readable vs. executable rules (continued) and enforcement records
Enforcement at the point of action also scales with the rule set. Members need not learn every rule before acting: when an edit violates a rule, the review refuses it and returns the reason, so the obligation surfaces when it becomes relevant (e.g., P01's refusal and compliant rewrite at step 205; see our response to Reviewer BuK6). Readable rules must instead be carried in every member's context in advance, a cost that grows with the number of rules an organization accumulates. Our runs involve a modest number of rules per organization, so we present this as design rationale rather than a measured result.
How reliable voluntary compliance would be if an LLM selected actions while reading the rules (the B1 regime) remains open, and we will list it as a limitation. Answering it would require 20+ additional B1-scale runs: LLM-selected action costs 45.2M tokens per run versus 7.3M for Relic (Table 1), while B1 has lower point estimates than Relic on all four verified endpoints. That cost is itself a reason long-running organizations need a cheaper decision layer, in which execution-time enforcement is how rules take effect. Within the channel that readable rules do reach, they help: Transfer-Text improves complete contracts over Fresh by 12.9 pp.
On complete contracts, Transfer Exec and Text differ by only +0.8 pp (32.3% vs. 31.5%), and we do not claim an executable-binding effect on this endpoint; the abstract will report it alongside the behavioral-case result. In transfer, the binding effect appears on held-out (+14.8 pp), exposed (+4.7 pp), confirmed-issue (+3.7 pp), and behavioral-case outcomes (+6.5 pp, 95% CI [0.7, 15.6]). On complete contracts, the binding evidence comes from the in-situ comparison (B3 vs. B3-text: +7.18 pp, 95% CI [3.54, 10.91]).
Enforcement records. We will rename the enforcement counts in Tables 31 and 46 and in Section 6 as protocol-linked enforcement records and report their composition. Across the 60 Terra+Opus B3 runs, the 33,443 records comprise 1,413 direct code-review refusals (4.2%), 28,997 records attributing shared CI outcomes to a protocol (86.7%), and 3,026 follow-up, release-gate, and dispute records; 7 records from one run retain only totals. These counts should not be read as actions blocked by learned rules alone.
Response to Reviewer XbQq: CooperBench, governance, ProgramBench
Q3: CooperBench.
- Exclusion set. Validity issues were admitted at the level of a feature or pair family, based only on a mismatch between the public task contract and the behavior the official evaluator requires (Table 56). Once a family was admitted, every exact pair containing it was excluded for all systems regardless of outcome; the exclusion set therefore also removes 21 pairs that Relic passed. The audit began partway through Relic execution, as a check that the task specifications were valid; it examined public specifications and evaluator behavior and did not consult any system's outcomes. Pairs from families admitted before Relic reached them (36 pairs) were excluded before Relic ran them, which is why they carry no Relic verdict. The audit was exhaustive over all 652 pairs, including pairs that Relic passed and baselines failed, and applied a single admission criterion, so the exclusion set does not depend on the order in which pairs were examined; each family's evidence (Table 56) can be checked without reference to outcomes. Ambiguity-aware families involve judgment, which is why we also report the definite-defect policy and the raw comparison. We will rewrite the sentence in M.1, make the raw matched comparison primary, and report both validity policies as sensitivity analyses. On the raw matched set, Relic is within 0.5 pp of Team no-proto (63.0% vs. 63.5%).
- Cross-model comparison. Released GPT-5.5-hao trajectories cover the fixed 48-pair subset, so we compare both models on identical pairs. Model capability does not uniformly favor Claude Opus 4.6: as a single agent it is slightly stronger (Solo 26/48 vs. 23/48), but in the two-peer topology that Relic uses, the official Opus baseline is much weaker (Peer 13/48 vs. 22/48). With the same model and the same two-peer split, Relic reaches 29/48, the highest count on this subset across both models and all released structures, including the full GPT-5.5 Team framework (26/48; one pair lacks an evaluation file and is counted as a failure) and Team without protocol (24/48). We acknowledge that part of the full-benchmark margin over GPT-5.5 Solo may reflect the single-agent capability gap. We did not rerun Solo, Peer, and Team with Claude Opus 4.6 on the full benchmark, or Relic with GPT-5.5-hao; the 48-pair subset provides the same-model comparison.
- Paired tests. Exact McNemar tests on the 48 pairs give p = 0.00015 for Relic vs. same-model Peer and p = 0.55 for Relic vs. same-model Solo. The recovery from the peer baseline is the finding on this subset; the margin over Solo is consistent with, but not established by, this subset.
Q4: Transfer sources and deadlock recovery. The concrete rules in the transfer package come from a single B3 organization (cattrs, W09, seed 1401), so 27 of the 30 targets run on other workloads; per-guard source protocols are listed in our response to Reviewer BuK6 (Q4). As the reviewer notes, the concerns to include were selected using protocol families recurring across source runs on W01–W10, so the targets are not fully held out at that level; target outcomes were not used in selection or normalization. Of the 280 formed protocols, 36 (12.9%) had at least one approval completed by bounded deadlock recovery at initial adoption; the remaining 87.1% were adopted entirely through member approvals.
Q5: ProgramBench. The baseline (64.164%) is the official mini-SWE-agent result with GPT-5.6 Sol at xhigh reasoning on the same 25 tasks; our protocol treatment uses the same model, reasoning setting, and agent configuration. programbench_pack_v0 was written by the authors, drawing on protocol families observed in Relic runs together with ProgramBench-specific guards (originality, probe protection); individual rules are not traced to specific source runs. This experiment therefore tests whether protocol-style executable constraints port to a single-agent harness, not autonomous protocol formation.
Claims in the abstract. We will revise the abstract to describe the mechanism as execution-time enforcement, report complete contracts alongside behavioral cases for transfer, report CooperBench raw and validity-filtered results with the model difference stated, and give the 48-pair comparison with its counts.
We hope these analyses address the soundness concerns. If the reviewer sees any remaining issue that the revisions above cannot resolve, we would be grateful to know which, so that we can address it directly.
Response to Reviewer XbQq: Seeded rules and how protocols act
We thank the reviewer for the detailed reading. Several points led us to correct our own description, and we revise the claims accordingly.
Q1: Seeded rules. The per-rule analysis across 87 of the 90 B3 endpoints is in our response to Reviewer 6SYn. In short: no recovered enforcement record is attributed to the two seeded rules; they are never revised; they are excluded from the code editor; no SDL feature reads them; and they generate no additional candidates. Their only identified influence is on later proposals, through the wish-clustering prompt and a reflection fallback path, which we disclose as a limitation. Learned review/merge lineages share the seeded rule's topic but not its content: the clustering prompt instructs the model not to restate adopted rules, and learned rules such as P01 impose more specific obligations (contract mapping with current smoke-test evidence). We did not run the additional B2-with-seeded or B3-without-seeded arms; the code-level analysis above traces the channels through which the seeded rules could act.
Q2: Readable vs. executable rules. We correct our own description. In native B3, adopted protocols do not enter SDL feature computation or change SDL weights; the SDL's protocol-related features depend only on action type and work state and are identical in B2 and B3. Adopted protocols act through three execution-side pathways: (i) they are readable in code-generation and reflection contexts; (ii) a protocol-aware code-change review refuses non-compliant edits and returns the rule-specific reason for revision; and (iii) outcomes of shared checks such as CI are attributed to the specific protocol they bear on, providing input to later reflection and amendment. In the Opus runs used for the B3-text comparison, pathway (ii) refused 452 distinct edit actions across 26 of 30 runs (1,149 refusal events), and 171 of those edits were then revised to comply. B3-text keeps pathway (i) and the full online learning process (reflection, wishes, proposals, governance, adoption, and revision), but disables (ii) and (iii), including the automatic attribution of CI outcomes to individual protocols; shared CI, review, and repository-validity checks continue to operate and report failures. In the transfer experiment, Exec's compiled guards additionally remove violating candidates before selection and re-check authorization before the handler runs; Text receives the same rule text without either check. Neither treatment changes the SDL scoring function, and the present data do not separate the contributions of the individual pathways.
This difference is the object of our claim rather than a confound. An organizational protocol is an obligation attached to a role, not a recommendation to whoever happens to read it. Its value is that compliance does not depend on each new occupant reading, interpreting, and choosing to follow the rule, which is exactly what changes under member turnover. Human software teams make the same choice: engineers know that changes need review and passing tests, yet teams install branch protection because voluntary compliance fails under deadline pressure and turnover.