acceptodds
Under review as a conference paper at ICLR 2027

Moral Hazard in Multi-Agent Language Models

Abstract

Cooperation can fail when costly, hard-to-observe effort benefits another agent. The Dialogue Moral Hazard Game instantiates this incentive problem for language agents: an agent chooses between a local reward and paying to reveal a hidden safety fact for a successor's decision. Across fourteen open-weight and four frontier models, we distinguish information acquisition, communication, downstream use, and team success. Private-share interventions reveal both accurate incentive-boundary tracking and safety-prioritizing departures despite correct payoff comprehension. Outcome-directed prompt optimization can bypass the cooperative mechanism: GEPA raises Muse Spark 1.1's team success from 22.2% to 100% while reducing querying from 51.1% to 0.3%. Freezing its prompts and changing the rank–label mapping reduces success to 12.5% and then 0%. We introduce CREDIT (Counterfactual Replay for Evidence-Driven Information Transfer), a mechanism-aligned multi-agent prompt-optimization algorithm using matched hidden-state twins, complete action replay, and staged causal credit. Across five models with three optimization seeds each, CREDIT increases realized information transfer relative to standard GEPA. Muse and Claude Opus sustain perfect query-mediated success across mappings; open-weight models expose remaining acquisition and downstream-use bottlenecks. These results distinguish successful outcomes from the information mechanisms that produce them.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent, llm, economics, cooperation, safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V4W6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper investigates moral hazard in multi-agent LLM systems, where an agent must incur a cost to perform an action that primarily benefits another agent. The authors introduce the Dialogue Moral Hazard Game, in which agents choose between completing a local task for a reward or paying to acquire information about an unsafe option in another agent's task, and they can then communicate this information to another agent (successor) to help them make a safe decision.

The authors evaluate 18 LLMs (14 open-weight and four frontier models), examining how agents respond to incentives and distinguishing between information acquisition, communication, downstream use, and team success. They find that agents can respond differently to private incentives, including prioritising another agent's safety despite understanding that doing so reduces their own reward.

The paper also investigates how prompt optimisation can achieve high team success without necessarily improving information transfer. The authors show that GEPA can learn to exploit a relationship between publicly observable information and hidden unsafe options, avoiding the need to query. They then introduce CREDIT, a counterfactual prompt-optimisation algorithm designed to encourage information acquisition, communication, and downstream use through matched hidden-state interventions and action replay. Across five models, CREDIT generally improves realised information transfer compared with standard GEPA, including under changes to the relationship between public utilities and hidden unsafe options.

Critical strengths
  1. The paper introduces an interesting framework for investigating costly cooperative behaviour in multi-agent LLM systems. The Dialogue Moral Hazard Game provides a controlled setting for examining situations in which agents must sacrifice their own reward to benefit another agent. The distinction between information acquisition, communication, downstream use, and team success is useful, as it enables the authors to investigate where cooperation succeeds or fails rather than evaluating only final outcomes. This provides a potentially valuable framework for understanding cooperation under conflicting incentives.

  2. The paper contains experimental and methodological contributions. The evaluation spans a relatively large number of models and includes controlled interventions and comparisons of different optimisation methods. In particular, the finding that a model can achieve high team success by exploiting a predictable relationship in the environment, rather than through information transfer, is interesting. The proposed CREDIT algorithm also offers a potentially useful approach to encouraging information-transfer mechanisms and evaluating their robustness under changes to the information structure.

Critical weaknesses
  1. The research questions and contributions are insufficiently motivated and connected, making it difficult to assess the significance of the work.

Although the abstract provides a relatively clear overview of the motivation, contributions, and main findings, this clarity was not maintained throughout the paper. Despite careful reading, it was difficult to understand how the different components of the research fit together and what the central scientific takeaway was.

The paper begins by examining how LLM agents respond to incentives when cooperative actions are costly, then investigates information-transfer mechanisms, before introducing CREDIT as a mechanism-aligned prompt-optimisation algorithm. However, the relationship between these different objectives, and why CREDIT is a necessary extension of the initial moral hazard investigation, is not sufficiently explained.

This is compounded by the presentation of the methodology. Important methods and concepts, such as GEPA, are referenced before being adequately introduced or motivated. The organisation also makes the paper difficult to follow, with experimental findings presented in Section 2 before the main Results section. The Discussion and Limitations section further provides little synthesis of what has been learned from the extensive experiments or how the findings advance our understanding of moral hazard in multi-agent LLM systems, as well as minimal discussion of the limitations.

The paper also attempts to cover several substantial contributions within a very limited main-text discussion, supported by an extensive appendix. While the breadth of the work is potentially a strength, I think the compression comes at a cost to clarity and coherence. Important methodological explanations, motivations, and experimental details receive insufficient attention in the main text, making it difficult to follow the progression of the research. The paper would benefit from a more focused narrative that prioritises clearly motivating and explaining its central contribution, rather than attempting to cover so many different aspects of the problem within the available space.

This is consequential to the overall assessment because, despite the potentially interesting technical and experimental contributions, the paper does not provide a sufficiently clear account of their significance. This made it difficult to reliably assess the overall contribution in its current form.

  1. The motivation for CREDIT's optimisation objective and the interpretation of its advantages over GEPA are not sufficiently established.

One of the main findings is that GEPA can achieve high team success without querying, by learning a relationship between publicly observable utilities and hidden unsafe options. However, in the original environment, this relationship is reliable, meaning that querying may not actually be necessary to achieve the optimal outcome. A policy that avoids querying can also obtain a higher game reward by avoiding its associated costs.

CREDIT instead explicitly encourages information-transfer mechanisms, including querying, through counterfactual supervision and stage-specific optimisation. While the paper demonstrates that this produces more robust query-mediated behaviour under changes to the information structure, it is not sufficiently clear when this mechanism should be preferred over policies that achieve the same outcomes without acquiring additional information. Indeed, the authors report cases where CREDIT achieves perfect querying and team success despite the positive net value of information falling to zero.

Furthermore, CREDIT uses substantially richer optimisation information than standard GEPA, including hidden-state twins, counterfactual action replay, and mechanism-specific validation. This makes it difficult to establish which elements of CREDIT are responsible for the improvements and to what extent the results demonstrate an advantage of the specific algorithm rather than its additional supervision and different optimisation objective.

These issues are important because they concern the motivation and interpretation of one of the paper's central methodological contributions.

Questions for authors
  1. What is the overarching research question, and how does it connect the incentive-response experiments, the investigation of information-transfer mechanisms, and the proposed CREDIT algorithm? What new understanding of moral hazard in multi-agent LLM systems emerges from these components collectively, beyond demonstrating that mechanism-aware optimisation can encourage the mechanism it explicitly rewards?

  2. What is the motivation for preferring query-mediated information transfer when a policy can achieve the same team outcome without querying, particularly given the associated costs? Under what conditions would CREDIT's preferred behaviour be considered better than a no-query policy, and how should the results where querying has no positive marginal value be interpreted?

  3. To what extent can the improvements over GEPA be attributed to CREDIT's specific counterfactual credit-assignment and staged optimisation procedure, rather than the additional information and supervision provided during optimisation? Are there ablations or comparisons under matched supervision that could help distinguish these effects?

8 comments

Thank you for taking the time to volunteer reviewing this paper. Your feedback has been very helpful.

We address your questions and comments below:

1. What is the overarching research question, how do the components connect, and what new understanding emerges beyond encouraging the mechanism explicitly rewarded?

Our overarching question was whether agents learned cooperation that generalized when acquiring information was privately costly and primarily benefited another agent. In doing so we were able to analyze how incentives and optimization affected their learning.

The three components addressed successive parts of this question:

  1. The incentive-response experiments tested DMHG’s construct validity by comparing agents’ query decisions with the theoretically derived incentive boundary (Figure 1). Some models correctly calculated that querying would reduce their own reward but still chose to query to help the other agent, while others more consistently chose the action that yielded the higher private reward.

  2. Then, we asked whether agents could learn, through optimization, to cooperate under those incentives. GEPA could increase team success while reducing querying (Section 4.2; Figure 3B), so improved outcomes alone did not answer that question. Freezing the prompts and changing the relationship between public utilities and unsafe options exposed policies that depended on predictive textual shortcuts rather than learned cooperation (Table 1; Appendix H). This revealed a further complication for moral hazard: optimization could change what agents predicted without paying for information, not merely their willingness to incur its cost. This was a surprising result.

  3. To address the predictive shortcuts, CREDIT was created by analyzing this failure. It varied hidden states while holding visible observations fixed, evaluated complete query and local-action alternatives, and trained acquisition, communication, and use before evaluating the complete interaction. This tested whether optimization could learn cooperation that remained effective when public predictors no longer sufficed (i.e if an optimized prompt can generalize).

CREDIT’s methodological contribution was a new prompt-update procedure defining how counterfactual evidence was constructed, how candidate policies were selected, and when training stages advanced. The matched-supervision comparisons (below in Question 3’s response) further tested how CREDIT’s components contribute to generalizable cooperation.

2. What is the motivation for preferring query-mediated information transfer when a policy can achieve the same team outcome without querying, particularly given the associated costs? Under what conditions would CREDIT’s preferred behaviour be considered better than a no-query policy, and how should results where querying has no positive marginal value be interpreted?

The motivation was to see if cooperative behavior can be learned, rather than a shortcut rule that produced successful outcomes only while public cues remained predictive. We tested this distinction, as we mentioned previously, by freezing the optimized prompts and changing which utility-ranked option was unsafe. Predictive GEPA strategies lost success when their learned rule stopped working, whereas CREDIT-trained strategies continued to acquire, communicate, and use case-specific information across the changed cue distributions. The intended improvement was learning cooperation that generalized.

Whether that generalization justified its costs was a separate economic question. The improvement in decisions and team outcomes had to exceed the query cost and forgone local reward. We examined this using the unchanged game reward, rather than CREDIT’s optimization score:

Rgame=0.35L‾+0.15F‾+0.50T−0.10Q‾.R_{\mathrm{game}} =0.35\overline L+0.15\overline F+0.50T-0.10\overline Q.

Thus, additional querying itself counted as a cost, not a benefit. For Gemma under the original mapping, CREDIT’s increases in final correctness and team success contributed +0.0236 and +0.0547 reward units. Reduced local reward and additional query cost contributed −0.0374 and −0.0216:

ΔRgame=0.0236+0.0547−0.0374−0.0216=+0.0193.\Delta R_{\mathrm{game}} =0.0236+0.0547-0.0374-0.0216 =+0.0193.

Its observed mean reward advantages were 0.0684 under the balanced mapping and 0.0934 under reversal.

For Opus under reversal, CREDIT’s policy achieved perfect success, but replacing one query with local work preserved success and earned at least as much reward. This showed that CREDIT’s successful cooperation did not mean every query justified its cost; those comparisons provided no economic reason to prefer the query.

3. To what extent can the improvements over GEPA be attributed to CREDIT’s specific counterfactual credit-assignment and staged optimisation procedure, rather than the additional information and supervision provided during optimisation? Are there ablations or comparisons under matched supervision that could help distinguish these effects?

Yes. We completed matched-supervision ablations, and CREDIT achieved higher observed mean team success and transfer even when the comparison method received the same feedback and staged starting prompts. We used four comparison conditions to distinguish additional training information from staged training and CREDIT’s joint selection procedure:

  1. We implemented GEPA-Twins in response to Reviewer iQnf’s Question 3, which requested training GEPA on “the same hidden-state twins using the ordinary game reward.” GEPA received CREDIT’s matched training worlds but retained ordinary trajectory feedback and game-reward selection, without replay feedback, explicit information targets, or staged training. This tested whether matching the training cases reproduced CREDIT’s improvement.

  2. We implemented GEPA-Replay to address Reviewer rgXo’s Weakness 3, which requested “GEPA trained with the same counterfactual data,” and Reviewer iQnf’s Question 3 about the contribution of replay. This condition received the same worlds as GEPA-Twins, plus trajectories from querying and each valid local-work alternative. Replacing a query removed its revealed information and regenerated affected communication and decisions. Game-reward selection remained unchanged, separating replay evidence from CREDIT’s additional targets and selection procedure.

  3. We implemented Target Feedback in response to Reviewer iQnf’s Question 3, which requested comparisons “adding target feedback and the curriculum separately.” This condition received GEPA-Replay’s training worlds and replay evidence, together with CREDIT’s explicit information targets and reward-difference diagnostics. These were supplied to the optimizer, while game-reward selection was retained without staged training. Comparing this with GEPA-Replay tested the additional target and diagnostic feedback.

  4. We implemented Staged Initialization in response to Reviewer iQnf’s Question 3, specifically the request to add “the curriculum separately.” This condition continued joint optimization from CREDIT’s saved first-three-stage prompts, retaining CREDIT’s feedback but selecting updates by ordinary game reward. Comparing it with Target Feedback tested staged training and its resulting initialization. Comparing it with CREDIT addressed your Question 3 about the specific algorithm’s contribution under matched supervision: feedback and staged starting prompts were matched, while joint selection and feasibility gates differed.

We evaluated these conditions under the balanced mapping, where public utility rank did not determine the unsafe option and query answers remained accurate. Results were mean ± sample SD across three optimization seeds, in percent.

Team Success

ModelGEPA-TwinsGEPA-ReplayTarget FeedbackStaged InitializationCREDIT
Qwen3-4B26.6 ± 18.423.4 ± 11.332.8 ± 9.542.2 ± 18.052.6 ± 22.8
OLMo-7B46.9 ± 8.739.1 ± 12.534.9 ± 5.944.3 ± 13.054.2 ± 12.5
Gemma-4B24.5 ± 2.427.1 ± 5.519.3 ± 3.634.4 ± 13.640.6 ± 8.3

Realized Information Transfer

ModelGEPA-TwinsGEPA-ReplayTarget FeedbackStaged InitializationCREDIT
Qwen3-4B0.5 ± 0.50.0 ± 0.022.9 ± 9.836.7 ± 11.352.1 ± 17.4
OLMo-7B16.4 ± 27.135.7 ± 23.05.7 ± 6.656.0 ± 25.673.4 ± 7.5
Gemma-4B6.3 ± 7.55.5 ± 3.44.9 ± 0.535.9 ± 23.557.0 ± 4.8

The comparisons showed that replay and additional target feedback did not consistently improve cooperative behaviour across these models. Adding staged initialization increased both outcomes on all three models. With feedback and staged initialization matched, CREDIT achieved a further 6.2–10.4 percentage points higher observed mean team success and 15.4–21.1 points higher transfer. This supports attribution from the staged optimization mechanism and CREDIT’s joint selection procedure beyond just additional information during optimization.

Reviewer_rgXoOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates moral hazard and information sharing in multi-agent language models. It introduces a controlled game to study costly information acquisition, communication, and downstream use, and proposes CREDIT, a counterfactual prompt-optimization framework intended to encourage reliable cooperation.

Critical strengths
  1. Interesting and meaningful research problem

    The paper investigates an important problem in multi-agent systems: whether successful outcomes genuinely reflect cooperation and information sharing. The distinction between outcome success and mechanism-level cooperation is insightful.

  2. Fine-grained behavioral diagnosis

    The paper decomposes cooperation into information acquisition, communication, and downstream information use, enabling a more systematic diagnosis of where collaboration fails. This is more informative than relying solely on aggregate task success and provides a useful experimental framework for studying multi-agent behaviors.

Critical weaknesses
  1. Dataset-induced shortcuts undermine the central conclusions

    The original environment deterministically makes the highest-utility option unsafe, allowing agents to infer the correct action without querying. Therefore, the no-query strategy learned by GEPA can be both effective and cost-efficient under the original distribution. Its failure under reversed or balanced mappings mainly demonstrates sensitivity to an altered data-generating rule, rather than necessarily indicating a failure of cooperation or moral-hazard mitigation.

  2. Mechanism compliance does not necessarily imply optimal cooperation

    CREDIT explicitly encourages information acquisition and transfer, but following this prescribed mechanism is not always economically beneficial. For example, Claude Opus achieves perfect query-mediated success while its positive net value-of-information rate drops to zero under the reversed mapping. This raises concerns about whether CREDIT learns cost-effective cooperation or simply reinforces a predefined workflow.

  3. Insufficient baselines and ablation studies

    CREDIT introduces matched hidden-state interventions, counterfactual replay, and staged optimization, but their individual contributions are not sufficiently isolated. Stronger comparisons are needed, particularly GEPA trained with the same counterfactual data, as well as simpler mechanism-aware prompting and reduced versions of CREDIT.

  4. Limited generalizability to realistic multi-agent settings

    The main experiments rely on a highly controlled synthetic environment with restricted information channels and action choices. Although additional strategic games are evaluated, they require task-specific prompt optimization. More realistic multi-agent tasks and non-deterministic information structures are needed to support the broader claims.

Questions for authors
  1. If the no-query strategy is optimal under the original data distribution, why should it be interpreted as a cooperation failure rather than efficient exploitation of available information?

  2. Can the authors evaluate non-deterministic information structures and compare CREDIT with simpler baselines under matched counterfactual supervision?

11 comments
AuthorsOpenReview ↗

Question 1

Thank you for volunteering your time to review our work. Your feedback was very beneficial and will be incorporated in a rebuttal revision of the paper. Please see responses to your questions below:

1. If the no-query strategy was optimal under the original distribution, why interpret it as a cooperation failure?

Additionally to address your critique in weakness 1, the cooperation failure was not avoiding an unnecessary query in the original distribution, per se. It was failing to optimize a prompt that robustly acquired, communicated, and used information across different unsafe-cue distributions. Several models, including frontier models, learned predictive rules that produced successful outcomes without learning a cooperative strategy. When the public cues no longer supported those rules, the predictive strategies failed without using the information-acquisition and communication route that remained available. Their original success therefore demonstrated successful prediction, not learned cooperation.

So the Dialogue Moral Hazard Game (DMHG) isolates two linked decisions: whether an agent would pay a query cost and forgo local reward to obtain information benefiting another agent, and whether the benefitting agent would forgo the highest-utility option when it was unsafe. The incentive-response experiments tested the first decision under a fixed information structure. The frozen-prompt mapping interventions then tested whether optimization had produced cooperation that generalized across unsafe-option distributions or a predictive rule fit to the original mapping.

The comparisons below included all three optimized prompts per model and method. Entries are GEPA / CREDIT team success, mean ± sample SD (%).

ModelOriginalBalancedReversed
Qwen3-4B IT15.6 ± 13.6 / 29.2 ± 12.515.4 ± 13.3 / 52.6 ± 22.816.9 ± 14.7 / 66.1 ± 21.3
OLMo-7B2.9 ± 3.7 / 42.2 ± 6.829.9 ± 3.5 / 54.2 ± 12.548.4 ± 4.3 / 52.1 ± 2.4
Gemma 4B15.1 ± 12.2 / 26.0 ± 8.920.6 ± 13.4 / 40.6 ± 8.324.5 ± 11.8 / 47.9 ± 6.3
Muse Spark 1.1100.0 ± 0.0 / 100.0 ± 0.012.5 ± 0.0 / 100.0 ± 0.00.0 ± 0.0 / 100.0 ± 0.0
Claude Opus 4.8100.0 ± 0.0 / 100.0 ± 0.041.7 ± 50.5 / 100.0 ± 0.033.3 ± 57.7 / 100.0 ± 0.0

The predictive GEPA strategies did not return to information sharing when their rule failed. Their original success, again, reflected successful prediction, not learned cooperation. CREDIT achieved higher mean success under both shifted mappings across all five models, including OLMo and Gemma, whose GEPA performance also improved under the shifts.

AuthorsOpenReview ↗

Question 2 part 1

2. Can CREDIT be evaluated with non-deterministic information and simpler baselines under matched counterfactual supervision?

Yes. We had already tested CREDIT under the balanced mapping, where public utility rank no longer determined which option was unsafe. We additionally tested an imperfect revelation experiment by evaluating frozen prompts when queries returned an incorrect unsafe-option label 25% of the time. We compared CREDIT with GEPA given the same training cases and outcomes of querying versus local work, simple instructions to query and share information, and CREDIT variants without staged training or contribution-based selection.

In the balanced test, we no longer always made the highest-utility option unsafe. It could be safe in one case and unsafe in another, so agents could not reliably identify the unsafe option from utility alone. They could still pay to learn which option was unsafe in another agent’s case and share that information. Queries gave correct answers in this test, allowing us to examine whether agents cooperated when the original predictive rule was no longer reliable.

We then tested whether cooperation persisted when queries themselves could be wrong, directly addressing Weakness 4’s request for non-deterministic information structures. We kept the optimized prompts frozen and made queries return the correct unsafe-option label 75% of the time and an incorrect label 25% of the time. Unlike the balanced test, where queries remained accurate, agents now had to communicate and act on potentially incorrect information without further optimization.

We addressed Weakness 3 with two sets of comparisons. First, we compared CREDIT with matched-training GEPA controls and simple fixed instructions under the 75% query-accuracy condition:

  1. GEPA-Twins received CREDIT’s matched hidden-state training worlds but retained ordinary trajectory feedback and game-reward selection. This tested the changed training distribution.

  2. GEPA-Replay additionally received query and valid local-action counterfactual trajectories, with affected communication and final decisions regenerated. Ordinary-reward selection was retained, addressing the request for GEPA trained with the same counterfactual data.

  3. Fixed instructions prescribed querying, communicating the unsafe label, and using the warning to choose the highest-utility remaining option. This addressed the request for simpler mechanism-aware prompting without optimization.

Results were mean ± sample SD, across three optimization seeds or three decoding repetitions for fixed instructions.

Team Success with 75% Query Accuracy (%)

ModelGEPAGEPA-TwinsGEPA-ReplayFixed InstructionsCREDIT
Qwen3-4B18.75 ± 16.3125.52 ± 15.6527.60 ± 5.4916.67 ± 0.9034.38 ± 18.42
OLMo-7B28.65 ± 3.2551.56 ± 6.8147.40 ± 15.9628.65 ± 0.9051.56 ± 1.56
Gemma-4B18.23 ± 10.4020.31 ± 2.7121.88 ± 4.1313.02 ± 3.2528.65 ± 11.73

Realized Information Transfer with 75% Query Accuracy (%)

ModelGEPAGEPA-TwinsGEPA-ReplayFixed InstructionsCREDIT
Qwen3-4B0.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.0037.24 ± 23.51
OLMo-7B8.85 ± 10.9514.84 ± 24.3729.95 ± 23.784.43 ± 3.1659.11 ± 2.51
Gemma-4B30.21 ± 17.372.34 ± 2.713.12 ± 2.8214.06 ± 2.3442.45 ± 7.90

CREDIT retained the highest observed mean transfer on all three models and the highest or tied mean team success. Its cooperation advantage therefore persisted when queries could return incorrect information, including against GEPA given counterfactual training outcomes and instructions explicitly prescribing the cooperative sequence.

AuthorsOpenReview ↗

Question 2 part 2

We also tested two reduced versions of CREDIT to determine whether its staged training and contribution-based selection improved cooperation:

  1. CREDIT without staged training skipped the initial stages that separately trained information acquisition, communication, and use. It instead began optimizing complete agent interactions using CREDIT’s final evaluation and selection procedure. This tested whether the initial training stages helped.

  2. CREDIT with game-reward selection kept the initial stages and the same feedback, but chose subsequent prompt updates by ordinary game reward instead of CREDIT’s contribution-based score. This tested whether CREDIT’s selection procedure helped after the initial training had been matched.

These were separate from the noisy-query experiment. We evaluated both variants under the balanced mapping, where utility rank did not determine the unsafe option, but query answers remained correct.

Results were mean ± sample SD across three optimization seeds (%).

Team Success

ModelWithout Staged TrainingGame-Reward SelectionCREDIT
Qwen3-4B47.4 ± 6.342.2 ± 18.052.6 ± 22.8
OLMo-7B50.5 ± 15.244.3 ± 13.054.2 ± 12.5
Gemma-4B33.9 ± 10.034.4 ± 13.640.6 ± 8.3

Realized Information Transfer

ModelWithout Staged TrainingGame-Reward SelectionCREDIT
Qwen3-4B26.6 ± 25.436.7 ± 11.352.1 ± 17.4
OLMo-7B67.2 ± 10.556.0 ± 25.673.4 ± 7.5
Gemma-4B39.6 ± 23.435.9 ± 23.557.0 ± 4.8

Normal CREDIT achieved higher observed mean team success and transfer than either reduced version on all three models. Both metrics were lower when the initial stages were skipped and when ordinary game reward replaced CREDIT’s contribution-based selection.

Finally, we addressed Weakness 4’s request for more realistic multi-agent tasks by optimizing CREDIT and GEPA in Melting Pot 2.0’s Clean Up task with Qwen3-4B IT. Seven agents collected apples for private reward, while cleaning pollution enabled future apple growth at the cost of time otherwise available for harvesting. Agents therefore faced repeated decisions about whether to harvest for themselves or maintain a resource that benefited others. This preserved DMHG’s privately costly cooperation problem but replaced its query–communication–decision sequence with a sequential decision dilemma.

To further test cooperation robustness, we froze the optimized prompts and changed the instructions given to zero to six partners. These partners used the same model but were instructed to prioritize personal harvesting, leave cleaning to others, and use ZAP to disrupt competitors. The focal agent and remaining partners retained their method’s frozen optimized prompt. Rewards, observation access, available actions, and simulator mechanics remained unchanged. This tested whether the learned policies sustained cooperation under a distribution shift in partner behavior, without further optimization.

Team reward was mean ± sample SD across three seeds, with 64 held-out episodes per seed and condition.

Aggressive PartnersGEPACREDIT
094.90 ± 83.26110.84 ± 73.21
171.99 ± 56.6390.88 ± 68.94
244.82 ± 33.5861.88 ± 60.17
334.57 ± 26.7035.56 ± 28.08
413.92 ± 11.6318.30 ± 20.78
56.41 ± 5.177.29 ± 7.35
63.22 ± 2.692.55 ± 1.47

CREDIT maintained higher observed mean team reward with up to five aggressive partners. When all six partners received aggressive instructions, both methods produced little reward, and CREDIT’s mean was lower. Its advantage therefore persisted across several partner-distribution shifts but did not extend to the most severe condition.

Reviewer_iQnfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies agent cooperation when gaining and sharing information beneficial to other agents are costly for the current agent. It constructs a game environment with costs and choices for each agent: it may choose to either earn a local reward or pay to reveal and share information beneficial to another agent. The paper target cooperartion incentive and proposes CREDIT, which uses matched hidden-state interventions, counterfactual replay, and staged prompt optimization to directly enhance cooperation. The paper separately evaluate the four ingredients of successful gameplay: information acquisition, communication, downstream use, and team success. Experiments over 18 models show that under certain experiment settings, prompt-optimized models achieve high scores by possibly learning the direct relationship between unrevealed labels and public utilities; on the other hand, CREDIT over 5 models improves realized information transfer and maintains team success over other experiment settings.

Critical strengths

The paper targets a meaningful question: which aspects in multi-agent cooperation drives successful teamplay. Over the designed game environment, the paper achieves this by separately evaluating information gathering, communication and use, and proposing the CREDIT framework. A good amount of experiment is done in the environment, helping to establish the proposed frameworks, analysis and contributions in depth.

Critical weaknesses
  1. The generalizability and broader implications of the framework and findings require stronger support. The game environment used by the paper poses a relatively small-scale, controlled task to each agent, while actual multi-LLM systems are increasingly being used in tasks in which information gathering, processing, and communication are complex. The design of CREDIT seemingly relies on specific traits of the game environment, such as exact information revelation and replayable downstream decisions. Some findings, including the emergence of rank-based shortcuts, are also closely connected to the specific game setting. The general insights these observations provide about multi-agent coordination, and the conditions under which they apply, therefore require clearer explanation. This limits the broader implications that can currently be drawn from the framework and findings.

  2. The contribution of CREDIT's individual components requires stronger experimental support. My understanding is that CREDIT changes the training distribution, provides additional counterfactual and target information, introduces a staged curriculum, and changes candidate selection. The comparison with standard GEPA therefore establishes the effect of this combined procedure. It remains unclear how much of the improvement requires the proposed replay and credit construction, which makes the additional value of CREDIT’s main components difficult to assess.

I currently lean towards weak reject. Despite the substantial controlled experiments, the broader implications of the findings remain insufficiently established; stronger evidence of applicability beyond the current controlled settings would substantially strengthen the paper. The additional value of CREDIT's main components is also not fully discussed, and an ablation study could help strengthen this part.

Questions for authors
  1. Clarification of the broader implications. What general insights do the findings, particularly the emergence of rank-based shortcuts, provide about multi-agent coordination, and under which conditions do these insights apply? How is the paper's buy-and-reveal setting relevant to real-world multi-LLM tasks involving more complex information gathering, processing, and communication?

  2. Clarifiction and evidence of generalizability. To what extent does CREDIT rely on exact information revelation and replayable downstream decisions? A clearer discussion of its applicability beyond the current game environment would help assess its generalizability. As additional experimental support, a few experiments illustrating and testing the framework and findings beyond the current controlled settings would substantially strengthen this aspect.

  3. Experimental evidence for the individual components. How much of CREDIT's improvement requires the proposed replay and credit construction? A particularly useful comparison would train GEPA on the same hidden-state twins using the ordinary game reward. Further comparisons adding target feedback and the curriculum separately would help establish the additional value of CREDIT. Can the authors provide these comparisons or evidence addressing this attribution?

5 comments

Thank you for volunteering to review and providing great feedback on the work. Please see responses to your questions below:

1. What general insights do the findings provide, under which conditions do they apply, and how is the setting relevant to complex multi-LLM tasks?

The central finding is that successful team outcomes can arise without agents acquiring, communicating, or using the information intended to support cooperation. Optimizing team success can instead learn a predictive relationship between public textual cues and the correct decision. These information strategies can produce identical successful outcomes but generalize differently when the cue–safety relationship changes. Particularly, fail to generalize.

The Dialogue Moral Hazard Game (DMHG) isolates an incentive problem: an agent pays a query cost and forgoes a local reward to acquire information that primarily benefits another agent. By making the highest-utility option unsafe in the original mapping, we can separately test whether the benefiting agent will forgo an attractive option when safety-relevant information requires it. In DMHG, we measured acquisition, communication, and downstream use to identify where cooperation succeeds or fails. These failures were heterogeneous across models, with bottlenecks arising at different stages in DMHG.

To test whether agents learned cooperation or a predictive heuristic that bypassed information acquisition, communication, and downstream use. We froze the optimized GEPA and CREDIT prompts to evaluate them on held-out distributions that changed the relationship between public utility ranks and the hidden unsafe options. The predictive GEPA policies failed when their learned rule no longer worked, without returning to the available cooperative route (i.e they do not generalize). Their original success therefore reflected successful prediction, not learned cooperation.

The following comparisons include all three optimized prompts per model and method. Entries are GEPA / CREDIT team success, mean ± sample SD (%).

ModelOriginalBalancedReversed
Qwen3-4B IT15.6 ± 13.6 / 29.2 ± 12.515.4 ± 13.3 / 52.6 ± 22.816.9 ± 14.7 / 66.1 ± 21.3
OLMo-7B2.9 ± 3.7 / 42.2 ± 6.829.9 ± 3.5 / 54.2 ± 12.548.4 ± 4.3 / 52.1 ± 2.4
Gemma 4B15.1 ± 12.2 / 26.0 ± 8.920.6 ± 13.4 / 40.6 ± 8.324.5 ± 11.8 / 47.9 ± 6.3
Muse Spark 1.1100.0 ± 0.0 / 100.0 ± 0.012.5 ± 0.0 / 100.0 ± 0.00.0 ± 0.0 / 100.0 ± 0.0
Claude Opus 4.8100.0 ± 0.0 / 100.0 ± 0.041.7 ± 50.5 / 100.0 ± 0.033.3 ± 57.7 / 100.0 ± 0.0

CREDIT achieves higher mean team success than GEPA under both shifted mappings across all five models. This includes OLMo and Gemma, whose GEPA performance also improves under the shifts. For OLMo, CREDIT achieves 54.2% versus 29.9% under the balanced mapping and 52.1% versus 48.4% under reversal. For Gemma, the corresponding comparisons are 40.6% versus 20.6% and 47.9% versus 24.5%. Muse and Opus CREDIT retain perfect success across all three mappings.

This finding is relevant to complex multi-LLM tasks when public cues change their relationship to the correct decision while an informative acquisition and communication route remains available. Information gathered by one agent must still be communicated and used by another.

2. To what extent does CREDIT rely on exact information revelation and replayable downstream decisions? What evidence supports applicability beyond the controlled game?

CREDIT used matched interventions and replayable downstream decisions during optimization, but its frozen policies did not require replay during execution. We found that perfect revelation was not necessary for its observed advantage over GEPA at evaluation.

To test dependence on exact revelation, we froze the optimized prompts and reduced query accuracy from 100% to 75%: each query had a 25% probability of returning incorrect information. Agents then communicated the returned information and made their downstream decisions without reoptimization for noisy answers. This directly tested whether CREDIT’s learned cooperation was robust imperfect information, rather than assuming every query revealed the correct information.

The following results are percentages, mean ± sample SD across three optimization seeds.

ModelGEPA Team SuccessCREDIT Team SuccessGEPA TransferCREDIT Transfer
Qwen3-4B18.75 ± 16.3134.38 ± 18.420.00 ± 0.0037.24 ± 23.51
OLMo-7B28.65 ± 3.2551.56 ± 1.568.85 ± 10.9559.11 ± 2.51
Gemma-4B18.23 ± 10.4028.65 ± 11.7330.21 ± 17.3742.45 ± 7.90

CREDIT retained higher observed mean team success and realized information transfer than standard GEPA on all three models.

To test whether CREDIT generalized beyond DMHG, we optimized CREDIT and GEPA in Melting Pot 2.0's Clean Up task with Qwen3-4B Instruct. In Clean UP, seven agents collect apples for individual reward, but can clean pollution to enable apples to continue to grow at the cost of time otherwise available for harvesting. Excessive pollution inhibits apple regrowth. This preserved the privately costly effort benefiting others that motivated DMHG, but replaced its short query–communication–decision sequence with a sequential decision dilemma involving repeated actions and delayed benefits.

Each episode lasts 1,000 environment steps per agent, with all seven agents using the optimized policy. Results were mean ± sample SD across three seeds, with 64 held-out episodes per seed.

MetricGEPACREDIT
Cumulative team reward94.90 ± 83.26110.84 ± 73.21
Pollution removed466.72 ± 9.24473.09 ± 22.07
Equality: 1−Gini1-\mathrm{Gini}0.567 ± 0.1470.594 ± 0.127

CREDIT achieved higher observed mean cumulative team reward, pollution removal, and equality (1−Gini1-\mathrm{Gini}) than GEPA.

To further test cooperation robustness, we then evaluated the GEPA and CREDIT prompts under a task distribution shift induced by increasing the number of aggressive partners. We froze the optimized prompts and assigned zero to six of the focal agent’s partners with aggressive instructions: “prioritize personal apple collection, leave cleaning to others, and use ZAP to temporarily remove competitors threatening access to apples”. These partners used the same model, while the focal agent and remaining partners retained their method’s frozen optimized prompt. These evaluations tested whether the learned policies sustained productive cooperation as more partners were instructed to exploit others’ maintenance efforts, without further optimization.

Team reward was mean ± sample SD across three seeds, with 64 held-out episodes per seed and condition.

Aggressive PartnersGEPACREDIT
094.90 ± 83.26110.84 ± 73.21
171.99 ± 56.6390.88 ± 68.94
244.82 ± 33.5861.88 ± 60.17
334.57 ± 26.7035.56 ± 28.08
413.92 ± 11.6318.30 ± 20.78
56.41 ± 5.177.29 ± 7.35
63.22 ± 2.692.55 ± 1.47

CREDIT maintained a higher mean team reward than GEPA with up to five aggressive partners. When all six partners received aggressive instructions, team reward fell sharply for both methods, and CREDIT’s mean was lower than GEPA’s. Its advantage therefore persisted across a range of partner-distribution shifts but not the most aggressive condition.

Further, we tested CREDIT’s generalizability across classic strategic games (Appendix M.3). We optimized prompts separately for each game using Qwen3-4B IT and OLMo-7B. In Sender–Receiver, OLMo CREDIT increased truthful reporting from 43.0 ± 21.9% to 63.3 ± 4.1% and improved private utility from −0.904 ± 0.052 to −0.487 ± 0.164, relative to GEPA. Qwen CREDIT and GEPA both achieved 100% efficient exchange and private utility 3.0 in Gift Exchange, and matched on team success, mechanism execution, and welfare in the social-objective Stag Hunt and Trust Game evaluations. These experiments extended the evaluation of CREDIT as a cooperation-optimization algorithm beyond DMHG’s specific information structure and action sequence.

3. How much improvement requires replay and credit construction? Can GEPA be compared on identical twins, with target feedback and the curriculum added separately?

Yes. We trained GEPA on identical hidden-state twins using ordinary game reward and separately evaluated replay, target feedback, and staged training. After feedback and staged initialization were matched, CREDIT’s joint selection yielded 6.2–12.5 percentage points higher observed mean team success across all nine model–mapping comparisons and 15.4–21.1 points higher balanced transfer.

We implemented:

  1. GEPA-Twins to address your explicit request for GEPA trained on “the same hidden-state twins using the ordinary game reward.” Visible observations remained identical while hidden unsafe options varied. Ordinary-reward selection tested whether the changed training distribution alone explained CREDIT’s improvement.

  2. GEPA-Replay to address your question about how much improvement required counterfactual replay. It added query and valid local-action trajectories, regenerating affected communication and final decisions, while retaining ordinary-reward selection. Comparing it with GEPA-Twins separated replay evidence from the matched training worlds.

  3. Target Feedback to address your request to add target feedback separately. Starting from GEPA-Replay, we added CREDIT’s information targets and net-value-of-information diagnostics to the proposer, retaining ordinary-reward selection without the curriculum. This separated the additional feedback from replay outcomes.

  4. A Staged Initialization control to address your request to evaluate the curriculum separately. We retained CREDIT’s acquisition, communication, and information-use stages, then performed joint optimization from their saved prompts using ordinary-reward selection and the same feedback. Comparing this control with Target Feedback tested staged training; comparing it with normal CREDIT tested contribution-based joint selection and feasibility gates after feedback and initialization had been matched.

Balanced-mapping results were mean ± sample SD across three optimization seeds, in percent.

Team Success

ModelGEPA-TwinsGEPA-ReplayTarget FeedbackStaged InitializationCREDIT
Qwen3-4B26.6 ± 18.423.4 ± 11.332.8 ± 9.542.2 ± 18.052.6 ± 22.8
OLMo-7B46.9 ± 8.739.1 ± 12.534.9 ± 5.944.3 ± 13.054.2 ± 12.5
Gemma-4B24.5 ± 2.427.1 ± 5.519.3 ± 3.634.4 ± 13.640.6 ± 8.3

Realized Information Transfer

ModelGEPA-TwinsGEPA-ReplayTarget FeedbackStaged InitializationCREDIT
Qwen3-4B0.5 ± 0.50.0 ± 0.022.9 ± 9.836.7 ± 11.352.1 ± 17.4
OLMo-7B16.4 ± 27.135.7 ± 23.05.7 ± 6.656.0 ± 25.673.4 ± 7.5
Gemma-4B6.3 ± 7.55.5 ± 3.44.9 ± 0.535.9 ± 23.557.0 ± 4.8

Replay and target feedback had mixed effects. Staged training increased balanced transfer on all three models, and CREDIT’s joint selection produced further gains after the feedback and initialization had been matched.

Notably, CREDIT had larger team success and information transfer than any of these alterations.

Thank you again for the insightful experiments. We will incorporate these results in a rebuttal revision of the paper.

Please let us know if there is anything we can address, clarify or elaborate for the revised paper that might lead you to augment your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.