- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies bargaining embedded in a long-horizon task. It models the environment as two linked stochastic games. A macro policy controls the outer game and decides when to start bargaining, and a separate bargaining policy makes and answers proposals. The agents are neural network policies trained from scratch with PPO in self-play. Bargaining uses structured propose, accept and counteroffer actions. Experiments use four-player Catan with two PPO league lineages. The update-450 agent beats common opponents 9 to 12 points more often than the update-50 agent. Policy swaps attribute this gain to the bargainer, while the macro policy shows no detectable improvement. Agreement rates and player-to-player trades rise. In a six-point variant, replacing the bridge with a full-state critic difference or with a completion bonus alone leads to near-universal acceptance and a win rate of 8.6% to 11.7%.
Critical strengths
When the value of a trade depends on the rest of the task, passing long-horizon value to a bargaining sub-policy is a real credit-assignment problem, and the paper states it clearly. The evaluation is statistically careful. All comparisons use common random numbers and paired bootstrap intervals, and the crossed policy-swap design attributes gains to specific components. The authors are candid about limitations, including that the intervals reflect gameplay noise rather than seed variance. The placement analysis and the record of failed training runs are useful diagnostics.
Critical weaknesses
W1. The auxiliary critic is trained on a reward that includes the potential , which is itself a hand-designed resource valuation based on build costs. Its inputs are also hand-picked resource and score features. The bridge signal may therefore largely approximate , which is a fixed pricing rule. This contradicts the claim of "no prescribed exchange values".
W2. The ablation uses a reduced six-point game, one seed per ablation arm, and only updates 50 and 100. The full-state arm is evaluated at update 50 only. The experiment is not enough.
W3. All arms share a completion bonus that is never defined. The collapse to near-universal acceptance is most directly explained by this bonus dominating the reward, not by the absence of the bridge. A no-bonus control is missing, and the development runs in A.13 use different settings and cannot replace it.
W4. There are no external opponents and no strong baselines. All opponents come from the same training population, with no fixed external agent such as a rule-based bot. The bargainer may simply exploit the acceptance habits of co-evolved opponents. The only bargainer baseline is a random bargainer, which the authors admit trades destructively.
W5. Novelty is limited and related work is missing. Rewarding a sub-policy with the change in a learned value function is common in hierarchical RL. In task-embedded negotiation, Diplomacy agents already evaluate agreements with learned value functions (Kramár et al., 2022; Meta FAIR, 2022), and the paper does not discuss them. The two-level stochastic-game formalism in Section 3 is not used for any result.
[1] János Kramár, Tom Eccles, Ian Gemp, Andrea Tacchetti, Kevin R. McKee, Mateusz Malinowski, Thore Graepel, and Yoram Bachrach. Negotiation and honesty in artificial intelligence methods for the board game of Diplomacy. Nature Communications, 13:7214, 2022. doi: 10.1038/s41467-022-34473-5.
[2] Meta Fundamental AI Research Diplomacy Team (FAIR), Anton Bakhtin, Noam Brown, et al. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378:1067–1074, 2022. doi: 10.1126/science.ade9097.
Questions for authors
Q1. What results do a -only bargaining reward and a bridge trained with give?
Q2. What is the completion bonus? How do the full method and the ablation arms behave without it? Do the full-state arm and the full method use the same normalization?
3 comments
Thank you again. The revised PDF now contains the experiments you asked for.
Q1 (potential-only reward; bridge with β = 0). At ten points (fresh seed per arm, through update 250), a bridge trained with no potential anywhere is level with the full league agent at update 250 (paired: −3.9 [−10.5, 2.7] and −2.0 [−8.2, 4.3]). A bargainer rewarded with the potential alone keeps pace through update 200 and then falls behind (−6.2 [−12.9, 0.4] and −7.4 [−12.9, −2.0]). Inside the fixed update-400 macro policy, the PBRS-only bargainer is behind the co-trained bargainer (−4.3 [−10.2, 1.6] and −9.4 [−15.2, −3.5]), while the bridge without PBRS is level (+5.1 [−0.4, 10.5] and −5.5 [−11.7, 0.8]). The learned signal does not need the potential, and the potential alone does not match it.
W1. Over 15,214 executed trades, the change in the potential explains 12% of the variance of the bridge signal, the best fixed per-resource price list 7%, and bank trades 2%; the two signals disagree in sign on 29% of trades, and the opponents' holdings contribute a non-zero term in 47%. The text now states precisely that the potential enters only the critic's training target, not the bargaining reward.
W3 / Q2. The completion bonus (0.005, one tenth of the normalized bridge reward's RMS target) and the shared normalizer are now defined in the appendix. A no-bonus arm wins 26.6% at update 50 against 25.4% for the full recipe.
W2 / W5. The six-point ablation now reports updates 50–150 for all arms, and the ten-point arms above extend it; one seed per arm is stated as a limitation, and the arms are continuing to update 400. Kramár et al. (2022) and FAIR (2022) are now discussed in Related Work.
We hope this addresses your main concerns, and would be grateful if you would consider them in your final assessment.
I would like to thank the additional experiment results provided by the authors. However, I believe the novelty is limited as stated by reviewer DQ1Q. I would like to see more realistic scenarios to be studied in the future version of the paper. Therefore, I remain my score.
Thank you for your excellent feedback -- we found many of your suggestions valuable and will work to incorporate them.
W1 / Q1 (bridge ≈ potential). Three points. (1) The two signals are weakly related. We measured this directly on the paper's own update-400 games (15,214 executed trades involving the evaluated agent). The change in the potential explains 12% of the variance of the bridge signal; the best possible fixed per-resource price list explains 7%; on bank trades, where the price is fixed by rule, those explain 2%. We also note that potential-based reward shaping (PBRS) enters the critic's target with a negative sign (a perfectly fit critic equals the win value minus 0.2Φ), so the correlation we measure is not an artifact of training on Φ. We will replace "no prescribed exchange values" a more precise statement: the bargainer's reward contains no fixed exchange values; the PBRS, derived from the build-cost table, contributes to the critic's training and explains a small share of the signal the bargainer sees. (2) The learned signal conditions on information the potential does not. The critic reads every player's holdings, so the reward for a trade includes what the trade does to the counterparty's position: that term is non-zero in 47% of the agent's trades and flips the sign of the reward in 6%. The critic's value also moves for third parties, whose own potential cannot move (median ∣\dv∣ 0.042 against 0.134 for the traders), and the identical exchange, three wheat for one ore, is valued anywhere from −0.06 to +0.29 depending on the state. A potential of the agent's own hand gives that exchange one value in every state. (3) The experiments you suggested. We are in the process of training a bargainer rewarded with the potential alone and a bridge with no potential anywhere in its training (one fresh seed each, the published budget) and evaluated both at updates 50 and 100 against the same control seeds. We will include their results in an updated PDF by 16 November and in the final version.
W3 / Q2 (completion bonus). The completion bonus is 0.005 per executed proposal. The bridge reward is normalized to an RMS of 0.05 and clipped at $\pm$0.15, so the bonus is one tenth of a typical bridge reward and one thirtieth of the largest. All arms, including the full-state arm, use the same normalizer and target. A no-bonus arm (fresh seed, same budget) wins 26.6% [21.5,32.3] at update 50 and 25.8% [20.8,31.5] at update 100, against the bridge's 25.4% and 27.3%, with a learned-minus-random contrast of +13.7 at both updates. The bonus is not the mechanism. The collapse of the sparse arm comes from the absence of a per-trade signal that discriminates between trades, not from a bonus rewarding any trade. The definition and the normalization will now be in the text.
W2 (ablation scope). The revised paper reports five arms, three of them on fresh seeds, at updates 50 and 100 (150 where available), including the two cells missing from the submission (sparse at update 150: 15.6% [11.7,20.6]; full-state at update 100: 7.8% [5.1,11.8]. One fresh seed per arm, matched to a three-seed control at the same update, and the six-point game remain limitations and are stated as such. Six points is where every arm could be trained from scratch and matched to that control; a ten-point arm costs about three weeks on five nodes. The ten-point arms mentioned above are the first step at closing that gap.
W4 (external opponents). We agree a fixed external opponent would strengthen the evaluation, and we do not have one: the available Catan bots (the JSettlers robot, Catanatron's players) live in other engines with different trading protocols and cannot be dropped into ours. We add this to limitations. Two observations bear on the concern that the bargainer exploits co-evolved acceptance habits. First, the Panel B opponents come from an independently seeded league run that the evaluated agent never trained against, and the bargainer effect there is the same size as against its own run (+23.4 and +25.8 points over a random bargainer). Second, the effect scales with the opponents' bargaining ability rather than being a fixed habit: against tables whose bargainers are early checkpoints the agent gains a further +10.9 [3.5,18.4] and +13.3 [6.6,20.3] points.
W5 (novelty, Diplomacy, Section 3). We now cite Kramar et al.\ (2022) and FAIR (2022). Both evaluate agreements with a full-state value model trained at large scale, the latter with human dialogue data. Our question is the complementary one: when the full-state value difference is too small to train a bargainer from scratch (our full-state arm: 7.8--8.6%, no better than random), a low-dimensional allocation-sensitive critic restores a usable signal. On Section 3: the bridge is the difference of the surrogate value across the bargaining subgame defined there; we now state the measured quantities in those terms and shorten the rest.