acceptodds
Under review as a conference paper at ICLR 2027

Competition Discovers, Cooperation Consolidates: Social Rewards Shape Self-Evolving LLM Agents

Abstract

Self-evolving LLM agents improve by converting interaction traces into persistent external memory. Yet while substantial effort has been devoted to refining downstream memory updaters, the upstream social environment that generates and filters these trajectories remains largely unexamined. In multi-agent self-evolution, social reward structures govern the evolutionary loop at two foundational stages: during interaction, process rewards dictate how agents search, communicate, and commit; after interaction, social credit assigns marginal value to outcomes, selecting which experiences are retained in persistent memory. To systematically explore this mechanism, we investigate two canonical social regimes: competition driven by relative individual credit, and cooperation driven by shared team outcomes. Our results reveal a fundamental division of labor: competitive pressure penalizes hesitation and incentivizes divergent exploration, accelerating environment-verified discovery under finite budgets and yielding decisive heuristics that elevate downstream speed-sensitive task completion. Cooperation instead aligns agents toward shared objectives and favors partner-legible proceduralization, producing structured routines that prove essential when downstream deployment requires inheriting complex collaborative conventions. Rather than revealing a universally superior paradigm, these findings show that competitive environments drive expansive frontier discovery, whereas cooperative environments synthesize inheritable order. Overall, social reward structures thus act not as incidental scaffolds, but as primary evolutionary drivers that determine both what multi-agent systems can discover and what persistent capabilities they ultimately inherit.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Social Reward, Competition, Cooperation, Multi-Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_e5zoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how competition and cooperation affect the self-evolution of LLM agents. It compares both the solutions discovered during interaction and the memories transferred to fresh agents, while keeping the memory updater fixed. Competition encourages exploration, whereas cooperation produces procedures that are easier for later agents to reuse. The paper also evaluates schedules that combine the two regimes.

Critical strengths
  1. I find the connection between diversity and memory inheritance interesting. A population can lose behavioral diversity while learning a procedure that improves subsequent agents. The paper examines this distinction through transfer, which gives more information than diversity or task accuracy alone.
  2. Evaluating fresh solvers with frozen memory separates inherited experience from the original interaction history. Section 6 provides memory controls and results across evolution lengths. The scheduling results also show that the preferred order depends on the model and task, rather than one schedule working best throughout.
Critical weaknesses
  1. The main claim concerns how diversity affects the experience inherited by later agents. The transfer experiments show that cooperative conventions can be reused, but say less about whether successors can depart from them when the conventions become unsuitable. This leaves a limit on how much the results tell us about avoiding diversity collapse. It does not undermine successful reuse in the tested tasks, and I would treat it as a boundary of the contribution rather than require diversity to remain high throughout.
  2. The experimental explanation based on terminal credit also needs clarification. Tables 2 and 5 give different AIME and HotpotQA contrasts. Table 5 distinguishes any-agent success from joint success that receives credit, but its relationship to Table 2 remains unclear. Reconciling the tables would help determine how much of the effect reflects improved task solving and how much reflects which successes receive credit.
Questions for authors
  1. Can successors recover when an inherited cooperative convention no longer fits the task?
  2. Can the terminal-credit effects in Tables 2 and 5 be reconciled under the same definition of task success?
4 comments

Response to Reviewer e5zo - Part (1/2) (Q1a)

Thank you very much for accurately summarizing our core thesis and for your thoughtful assessment of the scope of our contribution. Your intuition is spot on: whether successor agents can depart from obsolete conventions was indeed a key boundary of our initial analysis. Motivated by your comment, we conducted a dedicated set of experiments showing that this boundary can in fact be crossed—provided the training stages are ordered properly.


Q1: The main claim concerns how diversity affects the experience inherited by later agents. The transfer experiments show that cooperative conventions can be reused, but say less about whether successors can depart from them when the conventions become unsuitable. This leaves a limit on how much the results tell us about avoiding diversity collapse. It does not undermine successful reuse in the tested tasks, and I would treat it as a boundary of the contribution rather than require diversity to remain high throughout.

R1: We agree that this is an important question. Our new experiments demonstrate that as long as exploration precedes consolidation, successor agents can successfully depart from outdated conventions.

New Experiment: Can successors adopt a new convention after a convention reversal? Across 160 seeds in the emergent convention setup, we invert the color semantics in the second half of training and compare the frozen memories produced by five training schedules in terms of completion rate on 10 held-out circuits governed by the new rule (where yellow denotes the true breakpoint) versus the original rule:

Trained Memory RegimeNew-Rule CircuitsOriginal-Rule Circuits
No Memory0.3440.266
COOP-only (Full Cooperation)0.4130.814
COMP-only (Full Competition)0.3380.746
COOP→COMP Cooperate,thenCompetepost−reversalCooperate, then Compete post-reversal0.2540.877
COMP→COOP Compete,thenCooperatepost−reversalCompete, then Cooperate post-reversal0.6740.514
  • Exploring before consolidating enables escaping obsolete conventions: COMP→COOP (compete first, then cooperate after reversal) outperforms COOP-only by +0.261 (95% CI:
+01˙95,+03˙28+0\.195, +0\.328

; outperforming in 111 of 160 seeds and underperforming in 32).

  • Ordering is critical: The reverse schedule, COOP→COMP (cooperate first, then compete after reversal), offers no benefit and drops by −0.159 relative to COOP-only (
−02˙33,−00˙82−0\.233, −0\.082

). The gap between the two staged schedules reaches 0.420, with COMP→COOP prevailing in 121 out of 160 seeds.

  • Memory genuinely switches to the new rule: When evaluated on the original-rule circuits, the completion rate of the COMP→COOP memory drops to 0.514 comparedto08˙14forCOOP−onlycompared to 0\.814 for COOP-only, confirming that it replaces the outdated rule rather than retaining both rules simultaneously. Moreover, 52% of COMP→COOP memories explicitly record the color-meaning reversal, compared to only 29% under COOP-only.

This finding turns what was previously a boundary condition into a positive takeaway that aligns directly with Proposition C.5: exploration must precede consolidation; once a convention is crystallized into memory, switching to competition afterward cannot undo it.

Cross-Model Replication (Harder Circuits). To rule out the possibility that our conclusions depend on a specific circuit design or a single model, we constructed a harder set of circuits: decoys are placed along equally plausible alternative paths, and each marker is assigned a distinct color, driving the no-convention completion rate down to near-chance levels. We then replicated the convention-reversal experiment across 3 backbones. Averaged with equal weight across the 3 backbones, COMP→COOP competefirst,thencooperatepost−reversalcompete first, then cooperate post-reversal achieves a completion rate of 0.523, outperforming both COOP-only at 0.384 (+0.139, 95% CI:

00˙52,02˙240\.052, 0\.224

) and COOP→COMP cooperatefirst,thencompetepost−reversalcooperate first, then compete post-reversal at 0.396 (+0.127,

00˙33,02˙200\.033, 0\.220

. The per-backbone results completionrateonnew−rulecircuitscompletion rate on new-rule circuits are as follows:

BackboneNo MemoryCOOP-onlyCOMP-onlyCOOP→COMP post-rev\.COMP→COOP post-rev\.
gpt-5.40.2900.2650.3850.2300.575
gemini-3.7-flash0.5000.3800.5200.4350.560
claude-sonnet-4-60.3170.2920.3420.3000.358
Macro-Average0.4370.3840.4690.3960.523

Response to Reviewer e5zo - Part (2/2) (Q1b - Q2)

Q1 (continued)

Most backbones achieve their highest completion rate under COMP→COOP.

  • Frozen cooperative conventions are followed faithfully, even when they become misleading: in the emergent-convention experiment, swapping cyan and yellow inside an inherited cooperative memory drops completion from 0.814 to 0.367 no−memorybaseline:02˙66no-memory baseline: 0\.266, showing that successors execute the stored convention rather than correcting it. The new experiment clarifies the conditions under which agents can still break free from it.

  • However, conventions can still be revised online during active interaction: When the color rule is inverted during the final 4 of 12 rounds, cooperative pairs realign within a single round and solve 65% of the post-reversal rounds (62% overall), whereas competitive pairs solve only 13%.


Q2: The experimental explanation based on terminal credit also needs clarification. Tables 2 and 5 give different AIME and HotpotQA contrasts. Table 5 distinguishes any-agent success from joint success that receives credit, but its relationship to Table 2 remains unclear. Reconciling the tables would help determine how much of the effect reflects improved task solving and how much reflects which successes receive credit.

R2: Thank you for your careful check. Table 2 reports credit assignment, whereas Table 5 reports task success; Reading the two tables together is essential to disentangle task solving from credit attribution.

  • Table 2 reports Credit: The fraction of episodes in which an agent receives a positive terminal reward under a given credit assignment rule.

  • Table 5 reports Task Success: The fraction of episodes achieving any-agent success and joint success.

  • Joint Interpretation: The gap in Table 2 reflects how much additional credit is assigned under individual credit, whereas Table 5 reveals how much of that gap corresponds to genuinely solving the task.

Revision Plan: We will explicitly state the exact metric reported in each table's caption and place the credit and task success views side by side for direct comparison.

To further quantify run-to-run variance, we replicated the experiment across 16 random seeds ∗Any−Agent∗:atleastoneagentanswerscorrectly;∗Joint∗:bothagentsanswercorrectly;Becauseeachagent′sfinalanswerisscoredindividually,thetwocriteriacandivergeevenundersharedcredit*Any-Agent*: at least one agent answers correctly; *Joint*: both agents answer correctly; Because each agent's final answer is scored individually, the two criteria can diverge even under shared credit:

TaskBenchmark ChannelAny-Agent (Individual)Joint (Individual)Any-Agent (Shared)Joint (Shared)
AIMEBroadcast0.930.630.580.45
AIMEIsolated1.001.000.900.84
HotpotQABroadcast1.000.770.410.36
HotpotQAIsolated1.000.950.840.81
ToolUseBroadcast1.001.001.001.00
ToolUseIsolated1.001.001.001.00

Two key findings emerge from this replication:

  • First, the effect of credit assignment depends on the degree of trajectory coupling. Measured by any-agent accuracy, the gain of individual over shared credit varies substantially across communication channels: under a shared problem stream ∗Broadcast∗*Broadcast*, the gains are +0.35 (AIME) and +0.59 (HotpotQA), whereas under independent answering ∗Isolated∗*Isolated*, they drop to +0.10 and +0.16—an approximately 3.5×3.5\times difference. Reward structure begins to shape outcomes only when both agents operate along the same trajectory.

  • Second, shared credit suffers from a final-step bottleneck under a shared problem stream ∗∗∗∗∗Broadcast∗∗∗∗∗*****Broadcast*****. Performance is weakest in this setting 05˙8onAIME,04˙1onHotpotQA0\.58 on AIME, 0\.41 on HotpotQA and rebounds to 0.90 and 0.84 under independent answering ∗Isolated∗*Isolated*. In a shared problem stream, the team's outcome is dictated by the last-acting agent, so a mistake by either party penalizes the entire team—the tighter the coupling, the higher the coordination demand placed on cooperation.

Dear Reviewer e5zo,

Thank you for your review. Since the discussion time is less than a day, we would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Thank you for the clarifications and additional experiments. The rule-reversal experiment addresses my question about outdated conventions. It shows that competing first and cooperating after the reversal helps agents pass the updated rule on to their successors. The distinction between receiving positive reward in Table 2 and answering correctly in Table 5, together with the 16-seed results, also clarifies how credit assignment affects task performance. I maintain my Weak Accept rating.

Reviewer_wda4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how competitive and cooperative social regimes shape discovery and memory transfer in self-evolving LLM agents. With model parameters and the memory-update procedure fixed, it reports that competition promotes discovery while cooperation improves transferable procedural memory. It further explores staged transitions between the two regimes to combine their benefits.

Critical strengths
  • The overall design is simple and straightforward.
  • The paper is well-written.
  • The distinction between immediate discovery and frozen-memory transfer provides a clear structure for the empirical evaluation.
Critical weaknesses
  • The communication ablation also changes agent roles and the number of independently produced solutions. This makes the contribution of communication alone difficult to assess. The any-agent versus joint-success gap does not by itself demonstrate a reward-induced coordination effect. Since the updater receives regime labels, direct conditioning may also contribute to the observed memory differences.
  • The stated 120 runs do not readily reconcile with the reported episode counts and rates. For example, 68 successful episodes would imply 0.567 rather than 0.581 under this denominator. Tables 3 and 5 also show opposite credit-effect directions, despite being described as directionally consistent. Clarifying the aggregation rules and these task-dependent differences would improve confidence in the results.
  • The Minecraft transfer gains are promising, but the allocation of 100 circuits across five evolved memories is unclear. The statistical test and its treatment of dependence within each memory are also unspecified. More detail on the training–test split and successor backbones would clarify the scope of transfer. Per-memory results would help establish robustness across independent evolution runs.
  • The AI Scientist improvement from 1.4479 to 1.4452 is difficult to assess without seed-level uncertainty. The research budget and schedule-switching configuration also need further specification. The proposed diversity threshold of 0.40 would benefit from validation on held-out settings. Proposition C.3 provides a conditional diversity bound, but does not establish an advantage for staged scheduling.
Questions for authors
  • All the weaknesses above
9 comments

Response to Reviewer wda4 - Part (1/6) (Q1)

Thank you for your valuable feedback. Your comments focus on two core issues—whether our factor attribution is clean and whether our numerical results are internally consistent—which we address point by point below.

Q1: The communication ablation also changes agent roles and the number of independently produced solutions. This makes the contribution of communication alone difficult to assess. The any-agent versus joint-success gap does not by itself demonstrate a reward-induced coordination effect.

R1: Thank you for your careful reading and for pointing this out. We agree: Table 3 compares bundled protocol packages and cannot isolate communication alone; the clean isolation of the communication channel is provided in Table 5.

  • Table 3 (Holistic Protocol Bundle Comparison): Introducing the shared channel simultaneously introduces the diagnose→patch division of labor and changes the number of scorable patches from two to one. Because multiple factors shift together, Table 3 is not intended to isolate the contribution of the channel itself.

  • Table 5 (Channel-Only Isolation): Here, agent roles are held identical across groups, each agent still independently produces a solution, and prompts, rounds, seeds, and the updater remain fixed. The sole difference is whether the two agents share the same environment stream. All of our claims regarding the isolated effect of communication are grounded in Table 5.

To provide a multi-dimensional comparison, we replicated both the credit and accuracy analyses across 16 independent random seeds. The credit results are:

TaskShared CreditIndividual CreditGap (∆)
AIME0.4720.729+0.257
HotpotQA0.3610.865+0.504

The task success results (with Any-Agent and Joint defined as above) are:

TaskBenchmark ChannelAny-Agent (Individual)Joint (Individual)Any-Agent (Shared)Joint (Shared)
AIMEBroadcast0.930.630.580.45
AIMEIsolated1.001.000.900.84
HotpotQABroadcast1.000.770.410.36
HotpotQAIsolated1.000.950.840.81
ToolUseBroadcast1.001.001.001.00
ToolUseIsolated1.001.001.001.00

Crucially, the Channel ×\times Credit interaction dominates, making it insufficient to examine either main effect in isolation. Computing the difference-in-differences (DiD) on any-agent success yields:

TaskMain Effect of Credit OnlyMain Effect of Channel OnlyInteraction Term (DiD)
AIME+0.23+0.20+0.25
HotpotQA+0.38+0.22+0.43

The interaction term is comparable in magnitude to both main effects on AIME and exceeds both on HotpotQA. Consequently, the question of whether competition or cooperation is superior cannot be answered independently of the coupling structure; the impact of social incentives depends heavily on how tightly the two agents are coupled within the environment.

This directly reinforces our thesis that social structure is an upstream design variable: competition and cooperation are inherently composite regimes combining reward rules, information flow, and role structures. While factorial ablations can rule out individual components as sole drivers, the overall paradigm effect cannot be reduced to a single isolated factor—meaning credit assignment and interaction protocols must be co-designed.

Response to Reviewer wda4 - Part (2/6) (Q2 - Q3)

Q2: Since the updater receives regime labels, direct conditioning may also contribute to the observed memory differences.

R2: Thank you for this constructive comment. We acknowledge that the updater has access to regime labels, and we designed a dedicated control experiment specifically to bound this effect. Our conclusion is clear: the regime label itself does not produce observable differences in memory; Rather, the differences stem from the underlying interaction protocol.

Prompt-Only Control: Two memory updaters are fed the exact same raw training trajectory (identical execution traces and identical oracle evaluations), with the sole difference in their prompts being a single line describing the reward rule: one specifies "shared team completion reward," while the other specifies "individual race reward." Each updater distills a memory from this input, which is then frozen and evaluated with a fresh solver on held-out circuits. We chose this symmetric relabeling design rather than stripping the label entirely, as removing it would simultaneously remove the reward specification itself. Across 160 seeds, the two resulting memories show no difference in transfer performance: Cooperative−Competitive=−0.019\text{Cooperative} - \text{Competitive} = -0.019 (95% CI: [−0.085,+0.050][-0.085, +0.050]). When the reward description is the only difference, it confers no transfer advantage.

The complete results of this control are shown below (160 seeds, 480 held-out rollouts per condition, with seed-level bootstrap 95% CIs in brackets):

Fresh gpt-5.4 Solver with Frozen MemoryHeld-out Completion Rate [95% CI]
No Memory0.217 [0.181, 0.254]
Competitive Memory (Symmetric Prompt)0.315 [0.265, 0.367]
Cooperative Memory (Symmetric Prompt)0.296 [0.242, 0.350]
Cooperative Memory (Marker Semantics Swapped)0.067 [0.046, 0.092]

Both memories significantly outperform the no-memory baseline (Cooperative: +0.079 [0.017,0.142][0.017, 0.142]; Competitive: +0.098 [0.040,0.158][0.040, 0.158]), confirming that both capture transferable task knowledge. However, they are statistically indistinguishable from each other, demonstrating that conditioning on the regime label alone is insufficient to induce memory divergence.

  • Successor agents do not see regime labels at evaluation time either: When we swap the marker semantics inside the inherited cooperative memory, the completion rate plummets to 0.067—well below the 0.217 no-memory baseline—confirming that transfer is driven by the substantive content of the memory itself.

  • Divergence arises from the interaction protocol: When the two groups genuinely differ in their live interaction dynamics (shared team completion reward vs. individual race reward during actual co-play), the resulting memories diverge substantially (see R5 / Q5).


Q3: The stated 120 runs do not readily reconcile with the reported episode counts and rates. For example, 68 successful episodes would imply 0.567 rather than 0.581 under this denominator.

R3: Thank you for checking this carefully. The reported ratio is exact; the denominator in the text was a typo. That cell actually contains 117 episodes (13 tasks×3 seeds×3 episodes13 \text{ tasks} \times 3 \text{ seeds} \times 3 \text{ episodes}), where 68/117=0.581268 / 117 = 0.5812.

  • Revision Plan: We will update all occurrences of 120 to 117 in the text and correct the caption of Table 4 (which mistakenly stated 5 seeds instead of the actual 3).

Response to Reviewer wda4 - Part (3/6) (Q4 - Q5a)

Q4: Tables 3 and 5 also show opposite credit-effect directions, despite being described as directionally consistent. Clarifying the aggregation rules and these task-dependent differences would improve confidence in the results.

R4: Thank you for pointing this out. Below, we clarify how the tasks in the two tables differ and where the divergence arises:

  • Table 3 (Program Repair, pooled across 13 tasks): Comparing the two credit rules under the same shared channel, shared credit (0.658) outperforms individual credit (0.581). Breaking this down task by task in Table 4, shared credit is higher on 6 of the 13 tasks (ac3, crdt_merge, lr_parser, regex_engine, two_phase_commit, type_infer), individual credit is higher on 2, and the remaining 5 are tied. Under a leave-one-task-out check, the pooled gap remains stable between +0.055 and +0.092, confirming that this direction is not driven by any single outlier task.

  • Table 5 (AIME / HotpotQA, reported separately by benchmark, 10 held-out episodes per cell): Under the Broadcast channel, individual credit achieves higher any-agent success than shared credit.

Because the two tables differ in task domain, sample size, and unit of aggregation, we no longer frame them as mutually reinforcing evidence for a single uniform trend. Instead, we report them separately: shared credit prevails on multi-step program repair tasks, whereas individual credit prevails under the Broadcast channel on AIME and HotpotQA.

  • Revision Plan: We will remove the sentence claiming that the two tables "point in the same direction"; explicitly state the aggregation rule in the captions of both Table 3 (pooled across 13 tasks, cross-referencing the per-task breakdown in Table 4) and Table 5 (disaggregated by benchmark, 10 episodes per cell); and revise the main text to characterize the credit effects on the two task families separately.

Q5: The Minecraft transfer gains are promising, but the allocation of 100 circuits across five evolved memories is unclear. The statistical test and its treatment of dependence within each memory are also unspecified. More detail on the training–test split and successor backbones would clarify the scope of transfer. Per-memory results would help establish robustness across independent evolution runs.

R5: Thank you for these constructive comments. Regarding statistical testing, we perform paired tests at the seed level: multiple circuits evaluated under the same memory within a given seed are treated as repeated observations (clustered within the seed), and we report confidence intervals via seed-level bootstrap. Regarding circuit allocation, each of the 100 held-out circuits randomly loads one of the 5 evolved memories. Because these 5 memories originate from 5 independent evolution runs, memories and circuits are paired at random, and no memory is tuned toward any specific circuit. We will state this protocol explicitly in the revised manuscript.

What each reward regime writes into memory. Because 5 memories are too few to characterize the full distribution, we expanded our analysis to 100 evolved memories per reward regime and coded the content of each memory individually. Unlike the prompt-only control in R2, the two groups here differ in their actual live interaction protocol (shared team reward vs. individual race reward across 12 rounds of co-play), with no color legend provided at any point:

Memory ContentCooperative MemoryCompetitive Memory
States "cyan is the true breakpoint"98%94%
States "yellow is a decoy"97%84%
Notes that "color semantics reversed mid-training"39%12%
Records specific grid coordinates31%72%
Transfer completion rate (randomly sampled memory)0.8140.746
  • Agents discover the convention under both reward regimes (98% vs. 94% explicitly record the cyan rule), demonstrating that competition does not prevent agents from learning the underlying convention.

  • The key difference lies in whether memory encodes generalizable rules or episodic instances: Cooperative memories predominantly distill general rules, whereas 72% of competitive memories hard-code the specific coordinates of a single successful episode—more than double the rate under cooperation (31%). Specific coordinates fail to generalize when transferred to unseen circuit layouts.

  • Cooperative memories better capture non-stationary rule shifts: 39% of cooperative memories explicitly note that the color semantics reversed, compared to only 12% under competition. This aligns with our training-phase observation that cooperative pairs realign within a single round following a color reversal.

Reviewer_sTtTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how competition and cooperation shape self-evolving LLM-based agent systems. The authors treat social structure as an upstream variable that governs how experiences are generated and, in turn, which experiences enter long-term memory. They keep the backbone model parameters fixed and use the same Critic-Curate-Propose memory updater across all conditions. The experiments vary only the interaction protocols and credit assignment, with evaluation on eight benchmarks.

Critical strengths
  1. Most existing work on self-evolving systems focuses only on downstream memory updates. This paper instead examines how experiences are generated. Treating social structure as a design variable is a meaningful contribution to multi-agent system design.
  2. The overall research direction is sound. The authors keep the memory updater fixed and change only the social mechanism. They also use matched-protocol reward audits in Tables 2 and 5 and a 2-by-2 ablation of communication channel and credit assignment in Tables 3 and 4 to separate the effects of these two components.
Critical weaknesses
  1. The reward audit results are inconsistent. Table 2 reports AIME scores of 0.400 with shared credit and 0.700 with individual credit, a difference of +0.30. The corresponding HotpotQA scores are 0.350 and 0.850, a difference of +0.50. However, the "Broadcast" row in Table 5 and lines 1051 to 1052 report AIME scores of 0.80 and 1.00 (+0.20) and HotpotQA scores of 0.40 and 1.00 (+0.60). These values do not agree across the two tables.
  2. The takeaway in Section 7, lines 471 to 474, and the contributions paragraph claim that COMP-to-COOP outperforms single-mechanism approaches across the reported backbone and task combinations. Fig. 8(c) does not support this claim in all cases. For "claude-opus-5," the best schedule is COOP-to-COMP (0.001), while for "gemini-3.7-flash," it is "COMP only" (0.135). In Minecraft, COOP-to-COMP reaches 100%, compared with 80% for COMP-to-COOP. The claimed advantage of COMP-to-COOP therefore holds only in some settings.
Questions for authors

Which reward audit results are final, those in Table 2 or those in Table 5 and Appendix E.4? The reported AIME differences are +0.30 and +0.20, and the HotpotQA differences are +0.50 and +0.60, with discrepancies in the individual cells as well. What differences in the experimental setup account for these results?

3 comments

Response to Reviewer sTtT — Part 1/2 (Q1)

Thank you for your valuable feedback. Your observation regarding the apparent discrepancy between the tables helped us identify and resolve a persistent terminological ambiguity in our presentation of the metrics. Below is our point-by-point response.

Q1: The reward audit results are inconsistent. Table 2 reports AIME scores of 0.400 with shared credit and 0.700 with individual credit, a difference of +0.30. The corresponding HotpotQA scores are 0.350 and 0.850, a difference of +0.50. However, the "Broadcast" row in Table 5 and lines 1051 to 1052 report AIME scores of 0.80 and 1.00 +02˙0+0\.20 and HotpotQA scores of 0.40 and 1.00 +06˙0+0\.60. These values do not agree across the two tables.

R1: Thank you for this careful check, and we apologize for the lack of clarity in our original text. The two tables actually measure fundamentally different quantities and are therefore not expected to match numerically:

  • Table 2 reports Credit: the actual reward received by each agent under the given terminal-credit rule.

  • Table 5 reports Task Success: whether the task is genuinely solved, evaluated under both any-agent success and joint success criteria.

Specifically, credit is defined as the fraction of held-out episodes in which an agent receives a positive reward under a given credit assignment rule. Under shared credit, this is determined by the sign of the final-step reward and is identical for both agents; under individual credit, it is determined by the sign of each agent's cumulative score and may differ between agents. Because credit measures credit assignment rather than task accuracy, it operates on a different scale from Table 5 and should not be compared directly. In the revised manuscript, we will explicitly specify the exact metric reported in each table's caption.

To further quantify run-to-run variance and obtain deeper mechanistic observations, we replicated the experiments across 16 random seeds. The updated credit results are as follows:

TaskShared CreditIndividual CreditGap ∆∆
AIME0.4720.729+0.257
HotpotQA0.3610.865+0.504

The corresponding task success results are reported below Any-Agent: at least one of the two agents answers correctly; Joint: both agents answer correctly\. Because each agent's final answer is scored individually in these replications, the two criteria can diverge even under shared credit:

TaskBenchmark ChannelAny-Agent (Individual)Joint (Individual)Any-Agent (Shared)Joint (Shared)
AIMEBroadcast0.930.630.580.45
AIMEIsolated1.001.000.900.84
HotpotQABroadcast1.000.770.410.36
HotpotQAIsolated1.000.950.840.81
ToolUseBroadcast1.001.001.001.00
ToolUseIsolated1.001.001.001.00

Notably, this 16-seed evaluation reveals two key insights:

  • First, the effect of credit assignment depends heavily on the degree of trajectory coupling between agents. Measured by any-agent accuracy, the gain of individual over shared credit varies substantially across communication channels: under a shared problem stream (Broadcast), the gains are +0.35 (AIME) and +0.59 (HotpotQA), whereas under independent answering (Isolated), they drop to +0.10 and +0.16—an approximately 3.5×3.5\times difference. Reward structure meaningfully shapes outcomes only when both agents operate along a coupled trajectory.

  • Second, shared credit suffers from a final-step bottleneck under a shared problem stream (Broadcast). Performance is weakest in this regime 05˙8onAIME,04˙1onHotpotQA0\.58 on AIME, 0\.41 on HotpotQA, but rebounds to 0.90 and 0.84 when agents answer independently (Isolated). This occurs because the team outcome in a broadcast stream is dictated by the last-acting agent, meaning an error by either agent penalizes the entire team.

This further explains why merely modifying the reward or credit rule cannot substitute for a holistic competitive or cooperative social structure: isolated credit adjustments are heavily confounded by the underlying interaction topology, whereas our factorial analysis provides a more controlled characterization of the separate and interacting effects of credit assignment and communication channels.

Response to Reviewer sTtT — Part 2/2 (Q2)

Q2: The takeaway in Section 7, lines 471 to 474, and the contributions paragraph claim that COMP-to-COOP outperforms single-mechanism approaches across the reported backbone and task combinations. Fig. 8cc does not support this claim in all cases. For "claude-opus-5," the best schedule is COOP-to-COMP 00˙010\.001, while for "gemini-3.7-flash," it is "COMP only" 01˙350\.135. In Minecraft, COOP-to-COMP reaches 100%, compared with 80% for COMP-to-COOP. The claimed advantage of COMP-to-COOP therefore holds only in some settings.

R2: Thank you for this perceptive observation. Our intended takeaway is that **staged scheduling eg˙,˙COOP→COMPorCOMP→COOPe\.g\., COOP→COMP or COMP→COOP generally holds an advantage over single-mechanism regimes, **while the specific optimal ordering and magnitude of the gain depend on both the backbone and the task. Similarly, the fact that COOP→COMP achieves 100% vs. 80% for COMP→COOP on Minecraft further underscores that the optimal ordering is task-dependent.

To clarify this pattern systematically, we expanded our evaluation to include claude-4.5-sonnet and gpt-5-mini, bringing the total to 6 backbones alongside the original 4. The results reportedasheld−outerrorrate,wherelowerisbetterreported as held-out error rate, where lower is better are as follows:

Backbonecomp→coopcoop→compcomp-onlycoop-only
claude-opus-50.0070.0010.0050.011
gpt-5.6-sol0.0310.0460.0350.040
gpt-5.40.0090.0860.1330.013
gemini-3.7-flash0.1400.1570.1350.184
gpt-5-mini0.1340.1720.1360.121
claude-4.5-sonnet0.1560.1610.1270.139

This pattern reveals a capability threshold rather than a single universal winner:

  • Strong backbones: Staged scheduling is optimal. claude-opus-5 achieves the lowest error under COOP→COMP 0˙01\.001, whereas gpt-5.4 and gpt-5.6-sol perform best under COMP→COOP 0˙09and0˙31,respectively\.009 and \.031, respectively.

  • Weaker backbones: A single mechanism is optimal. gemini-3.7-flash 1˙35\.135, gpt-5-mini 1˙21\.121, and claude-4.5-sonnet 1˙27\.127 each favor a single-mechanism regime, and which single mechanism prevails also varies across models.

  • Consistent with Section 7: The optimal schedule reflects underlying backbone capability; staged scheduling yields gains only when a backbone is sufficiently capable of both exploring effectively and consolidating its discoveries.

In the revised manuscript, we will tone down the overgeneralized claim that COMP→COOP universally outperforms single mechanisms across all backbones and tasks, and instead refine our statement to: staged scheduling outperforms any single mechanism on sufficiently capable backbones, whereas initial diversity may provide a useful heuristic for choosing the ordering.

Dear Reviewer sTtT,

Thank you for your review. Since the discussion time is less than a day, we would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Comments on the paper

1 comment

General Author Rebuttal

We sincerely thank all reviewers for their insightful and thoughtful feedback.

  • Significance of the Research Perspective. We are deeply encouraged that the reviewers recognize our shift in focus beyond downstream memory updates to investigating how experience itself is generated, treating social structure as an explicit design variable in multi-agent systems (Reviewers #sTtT, #e5zo).

  • Transfer Evaluation under Frozen Memory. We are also pleased that the reviewers appreciate our core evaluation design—evaluating fresh solvers equipped with frozen memory to decouple inherited experience from raw interaction history—which yields richer insights than examining diversity or task accuracy alone (Reviewers #wda4, #e5zo).

  • Experimental Rigor and Scheduling Takeaways. We further appreciate the reviewers' positive assessment of our experimental controls and clarity of presentation, including fixing the updater while varying only the social mechanism, the matched-protocol reward audit, and the 2×22\times 2 ablation study (Reviewers #sTtT, #wda4), as well as our scheduling takeaway that the optimal ordering is both model- and task-dependent (Reviewer #e5zo).

In response to the reviewers' comments, we have conducted additional experiments and analyses, including: 11 a 16-seed replication of the credit audit, 22 160-seed symmetric-prompt replication and emergent convention experiments, 33 an "escaping" experiment following convention reversal, replicated across 4 backbones on harder circuits, and 44 Proposition C.5 complete with its full formal proof and numerical verification. We once again thank all three reviewers for helping us strengthen our work, and we will incorporate all feedback into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.