- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper investigates moral hazard in multi-agent LLM systems, where an agent must incur a cost to perform an action that primarily benefits another agent. The authors introduce the Dialogue Moral Hazard Game, in which agents choose between completing a local task for a reward or paying to acquire information about an unsafe option in another agent's task, and they can then communicate this information to another agent (successor) to help them make a safe decision.
The authors evaluate 18 LLMs (14 open-weight and four frontier models), examining how agents respond to incentives and distinguishing between information acquisition, communication, downstream use, and team success. They find that agents can respond differently to private incentives, including prioritising another agent's safety despite understanding that doing so reduces their own reward.
The paper also investigates how prompt optimisation can achieve high team success without necessarily improving information transfer. The authors show that GEPA can learn to exploit a relationship between publicly observable information and hidden unsafe options, avoiding the need to query. They then introduce CREDIT, a counterfactual prompt-optimisation algorithm designed to encourage information acquisition, communication, and downstream use through matched hidden-state interventions and action replay. Across five models, CREDIT generally improves realised information transfer compared with standard GEPA, including under changes to the relationship between public utilities and hidden unsafe options.
Critical strengths
-
The paper introduces an interesting framework for investigating costly cooperative behaviour in multi-agent LLM systems. The Dialogue Moral Hazard Game provides a controlled setting for examining situations in which agents must sacrifice their own reward to benefit another agent. The distinction between information acquisition, communication, downstream use, and team success is useful, as it enables the authors to investigate where cooperation succeeds or fails rather than evaluating only final outcomes. This provides a potentially valuable framework for understanding cooperation under conflicting incentives.
-
The paper contains experimental and methodological contributions. The evaluation spans a relatively large number of models and includes controlled interventions and comparisons of different optimisation methods. In particular, the finding that a model can achieve high team success by exploiting a predictable relationship in the environment, rather than through information transfer, is interesting. The proposed CREDIT algorithm also offers a potentially useful approach to encouraging information-transfer mechanisms and evaluating their robustness under changes to the information structure.
Critical weaknesses
- The research questions and contributions are insufficiently motivated and connected, making it difficult to assess the significance of the work.
Although the abstract provides a relatively clear overview of the motivation, contributions, and main findings, this clarity was not maintained throughout the paper. Despite careful reading, it was difficult to understand how the different components of the research fit together and what the central scientific takeaway was.
The paper begins by examining how LLM agents respond to incentives when cooperative actions are costly, then investigates information-transfer mechanisms, before introducing CREDIT as a mechanism-aligned prompt-optimisation algorithm. However, the relationship between these different objectives, and why CREDIT is a necessary extension of the initial moral hazard investigation, is not sufficiently explained.
This is compounded by the presentation of the methodology. Important methods and concepts, such as GEPA, are referenced before being adequately introduced or motivated. The organisation also makes the paper difficult to follow, with experimental findings presented in Section 2 before the main Results section. The Discussion and Limitations section further provides little synthesis of what has been learned from the extensive experiments or how the findings advance our understanding of moral hazard in multi-agent LLM systems, as well as minimal discussion of the limitations.
The paper also attempts to cover several substantial contributions within a very limited main-text discussion, supported by an extensive appendix. While the breadth of the work is potentially a strength, I think the compression comes at a cost to clarity and coherence. Important methodological explanations, motivations, and experimental details receive insufficient attention in the main text, making it difficult to follow the progression of the research. The paper would benefit from a more focused narrative that prioritises clearly motivating and explaining its central contribution, rather than attempting to cover so many different aspects of the problem within the available space.
This is consequential to the overall assessment because, despite the potentially interesting technical and experimental contributions, the paper does not provide a sufficiently clear account of their significance. This made it difficult to reliably assess the overall contribution in its current form.
- The motivation for CREDIT's optimisation objective and the interpretation of its advantages over GEPA are not sufficiently established.
One of the main findings is that GEPA can achieve high team success without querying, by learning a relationship between publicly observable utilities and hidden unsafe options. However, in the original environment, this relationship is reliable, meaning that querying may not actually be necessary to achieve the optimal outcome. A policy that avoids querying can also obtain a higher game reward by avoiding its associated costs.
CREDIT instead explicitly encourages information-transfer mechanisms, including querying, through counterfactual supervision and stage-specific optimisation. While the paper demonstrates that this produces more robust query-mediated behaviour under changes to the information structure, it is not sufficiently clear when this mechanism should be preferred over policies that achieve the same outcomes without acquiring additional information. Indeed, the authors report cases where CREDIT achieves perfect querying and team success despite the positive net value of information falling to zero.
Furthermore, CREDIT uses substantially richer optimisation information than standard GEPA, including hidden-state twins, counterfactual action replay, and mechanism-specific validation. This makes it difficult to establish which elements of CREDIT are responsible for the improvements and to what extent the results demonstrate an advantage of the specific algorithm rather than its additional supervision and different optimisation objective.
These issues are important because they concern the motivation and interpretation of one of the paper's central methodological contributions.
Questions for authors
-
What is the overarching research question, and how does it connect the incentive-response experiments, the investigation of information-transfer mechanisms, and the proposed CREDIT algorithm? What new understanding of moral hazard in multi-agent LLM systems emerges from these components collectively, beyond demonstrating that mechanism-aware optimisation can encourage the mechanism it explicitly rewards?
-
What is the motivation for preferring query-mediated information transfer when a policy can achieve the same team outcome without querying, particularly given the associated costs? Under what conditions would CREDIT's preferred behaviour be considered better than a no-query policy, and how should the results where querying has no positive marginal value be interpreted?
-
To what extent can the improvements over GEPA be attributed to CREDIT's specific counterfactual credit-assignment and staged optimisation procedure, rather than the additional information and supervision provided during optimisation? Are there ablations or comparisons under matched supervision that could help distinguish these effects?
8 comments
2. What is the motivation for preferring query-mediated information transfer when a policy can achieve the same team outcome without querying, particularly given the associated costs? Under what conditions would CREDIT’s preferred behaviour be considered better than a no-query policy, and how should results where querying has no positive marginal value be interpreted?
The motivation was to see if cooperative behavior can be learned, rather than a shortcut rule that produced successful outcomes only while public cues remained predictive. We tested this distinction, as we mentioned previously, by freezing the optimized prompts and changing which utility-ranked option was unsafe. Predictive GEPA strategies lost success when their learned rule stopped working, whereas CREDIT-trained strategies continued to acquire, communicate, and use case-specific information across the changed cue distributions. The intended improvement was learning cooperation that generalized.
Whether that generalization justified its costs was a separate economic question. The improvement in decisions and team outcomes had to exceed the query cost and forgone local reward. We examined this using the unchanged game reward, rather than CREDIT’s optimization score:
Thus, additional querying itself counted as a cost, not a benefit. For Gemma under the original mapping, CREDIT’s increases in final correctness and team success contributed +0.0236 and +0.0547 reward units. Reduced local reward and additional query cost contributed −0.0374 and −0.0216:
Its observed mean reward advantages were 0.0684 under the balanced mapping and 0.0934 under reversal.
For Opus under reversal, CREDIT’s policy achieved perfect success, but replacing one query with local work preserved success and earned at least as much reward. This showed that CREDIT’s successful cooperation did not mean every query justified its cost; those comparisons provided no economic reason to prefer the query.
3. To what extent can the improvements over GEPA be attributed to CREDIT’s specific counterfactual credit-assignment and staged optimisation procedure, rather than the additional information and supervision provided during optimisation? Are there ablations or comparisons under matched supervision that could help distinguish these effects?
Yes. We completed matched-supervision ablations, and CREDIT achieved higher observed mean team success and transfer even when the comparison method received the same feedback and staged starting prompts. We used four comparison conditions to distinguish additional training information from staged training and CREDIT’s joint selection procedure:
-
We implemented GEPA-Twins in response to Reviewer iQnf’s Question 3, which requested training GEPA on “the same hidden-state twins using the ordinary game reward.” GEPA received CREDIT’s matched training worlds but retained ordinary trajectory feedback and game-reward selection, without replay feedback, explicit information targets, or staged training. This tested whether matching the training cases reproduced CREDIT’s improvement.
-
We implemented GEPA-Replay to address Reviewer rgXo’s Weakness 3, which requested “GEPA trained with the same counterfactual data,” and Reviewer iQnf’s Question 3 about the contribution of replay. This condition received the same worlds as GEPA-Twins, plus trajectories from querying and each valid local-work alternative. Replacing a query removed its revealed information and regenerated affected communication and decisions. Game-reward selection remained unchanged, separating replay evidence from CREDIT’s additional targets and selection procedure.
-
We implemented Target Feedback in response to Reviewer iQnf’s Question 3, which requested comparisons “adding target feedback and the curriculum separately.” This condition received GEPA-Replay’s training worlds and replay evidence, together with CREDIT’s explicit information targets and reward-difference diagnostics. These were supplied to the optimizer, while game-reward selection was retained without staged training. Comparing this with GEPA-Replay tested the additional target and diagnostic feedback.
-
We implemented Staged Initialization in response to Reviewer iQnf’s Question 3, specifically the request to add “the curriculum separately.” This condition continued joint optimization from CREDIT’s saved first-three-stage prompts, retaining CREDIT’s feedback but selecting updates by ordinary game reward. Comparing it with Target Feedback tested staged training and its resulting initialization. Comparing it with CREDIT addressed your Question 3 about the specific algorithm’s contribution under matched supervision: feedback and staged starting prompts were matched, while joint selection and feasibility gates differed.
We evaluated these conditions under the balanced mapping, where public utility rank did not determine the unsafe option and query answers remained accurate. Results were mean ± sample SD across three optimization seeds, in percent.
Team Success
| Model | GEPA-Twins | GEPA-Replay | Target Feedback | Staged Initialization | CREDIT |
|---|---|---|---|---|---|
| Qwen3-4B | 26.6 ± 18.4 | 23.4 ± 11.3 | 32.8 ± 9.5 | 42.2 ± 18.0 | 52.6 ± 22.8 |
| OLMo-7B | 46.9 ± 8.7 | 39.1 ± 12.5 | 34.9 ± 5.9 | 44.3 ± 13.0 | 54.2 ± 12.5 |
| Gemma-4B | 24.5 ± 2.4 | 27.1 ± 5.5 | 19.3 ± 3.6 | 34.4 ± 13.6 | 40.6 ± 8.3 |
Realized Information Transfer
| Model | GEPA-Twins | GEPA-Replay | Target Feedback | Staged Initialization | CREDIT |
|---|---|---|---|---|---|
| Qwen3-4B | 0.5 ± 0.5 | 0.0 ± 0.0 | 22.9 ± 9.8 | 36.7 ± 11.3 | 52.1 ± 17.4 |
| OLMo-7B | 16.4 ± 27.1 | 35.7 ± 23.0 | 5.7 ± 6.6 | 56.0 ± 25.6 | 73.4 ± 7.5 |
| Gemma-4B | 6.3 ± 7.5 | 5.5 ± 3.4 | 4.9 ± 0.5 | 35.9 ± 23.5 | 57.0 ± 4.8 |
The comparisons showed that replay and additional target feedback did not consistently improve cooperative behaviour across these models. Adding staged initialization increased both outcomes on all three models. With feedback and staged initialization matched, CREDIT achieved a further 6.2–10.4 percentage points higher observed mean team success and 15.4–21.1 points higher transfer. This supports attribution from the staged optimization mechanism and CREDIT’s joint selection procedure beyond just additional information during optimization.
Thank you for taking the time to volunteer reviewing this paper. Your feedback has been very helpful.
We address your questions and comments below:
1. What is the overarching research question, how do the components connect, and what new understanding emerges beyond encouraging the mechanism explicitly rewarded?
Our overarching question was whether agents learned cooperation that generalized when acquiring information was privately costly and primarily benefited another agent. In doing so we were able to analyze how incentives and optimization affected their learning.
The three components addressed successive parts of this question:
The incentive-response experiments tested DMHG’s construct validity by comparing agents’ query decisions with the theoretically derived incentive boundary (Figure 1). Some models correctly calculated that querying would reduce their own reward but still chose to query to help the other agent, while others more consistently chose the action that yielded the higher private reward.
Then, we asked whether agents could learn, through optimization, to cooperate under those incentives. GEPA could increase team success while reducing querying (Section 4.2; Figure 3B), so improved outcomes alone did not answer that question. Freezing the prompts and changing the relationship between public utilities and unsafe options exposed policies that depended on predictive textual shortcuts rather than learned cooperation (Table 1; Appendix H). This revealed a further complication for moral hazard: optimization could change what agents predicted without paying for information, not merely their willingness to incur its cost. This was a surprising result.
To address the predictive shortcuts, CREDIT was created by analyzing this failure. It varied hidden states while holding visible observations fixed, evaluated complete query and local-action alternatives, and trained acquisition, communication, and use before evaluating the complete interaction. This tested whether optimization could learn cooperation that remained effective when public predictors no longer sufficed (i.e if an optimized prompt can generalize).
CREDIT’s methodological contribution was a new prompt-update procedure defining how counterfactual evidence was constructed, how candidate policies were selected, and when training stages advanced. The matched-supervision comparisons (below in Question 3’s response) further tested how CREDIT’s components contribute to generalizable cooperation.