acceptodds
Under review as a conference paper at ICLR 2027

Bluff Trade: A Multi-Agent Benchmark for Multi Turn Bluffing and Bidding

Abstract

We introduce BLUFF TRADE, a multi-agent benchmark for evaluating large language models (LLMs) as agents in strategic reasoning under imperfect information, adversarial interaction, and resource constraints. The benchmark combines auctions, hidden-offer trade challenges (TCs), bargaining, bluffing, opponent modeling, and resource allocation within a single long-horizon game lasting 50–60 turns. Unlike prior agent benchmarks that test these abilities in isolation, BLUFF TRADE evaluates whether agents integrate them across a competitive, multi-agent economic game with conflicting incentives. The benchmark logs every bid, TC offer, counteroffer, and card selection, enabling behavioural analysis beyond final scores or win rates. We evaluate several language models amogst each other, fixed code policies and neural networks trained using deep reinforcement learing. Strategic coherence, in particular spending efficiency, resource discipline, and phase-adaptive bidding, is associated with rank more strongly than spending volume or any single subskill. Two heuristic code agents outperform most tested LLMs, and behavioural traces surface recurring LLM failure modes including overbidding, self-bidding, bankrupt TC initiation, and weak opponent-state adaptation. Evaluating agentic competence requires benchmarks that test the joint deployment of multiple capabilities in multi-agent environments with conflicting incentives, uncertainty, andeconomic dynamics.

Primary area:
datasets and benchmarks
Keywords:
LLM Eval, Strategic Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rSZNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The work proposes Cattle Trade, a benchmark based on a four-player version of the existing trading card game Kuhhandel, combining bargaining, deception, and resource constraints. The evaluation includes eight LLMs (three tested at both low and high reasoning effort) and three rule-based agents, resulting in 14 player configurations. Each configuration plays 52 games, with 182 games in total. The paper provides quantitative win metrics as well as qualitative analysis.

Critical strengths
  1. The work studies frontier models like Opus 5.5, Fable 5.1 and Sol 6, which were released closer to the paper submission deadline.
  2. Qualitative analysis is good. The paper discusses spending, trade activity, bluffing, and specific rule-understanding mistakes.
  3. Implementation details provided in the appendix are useful for reproducibility.
Critical weaknesses
  1. The additional value of this benchmark is not sufficiently established. Table 1 distinguishes Cattle Trade from other benchmarks through different parameters but does not highlight what practical problem the benchmark addresses. For example, Andon Labs's Vending Bench Arena [1] already evaluates frontier models on simulations involving competing economic agents.

  2. Win rate depends on the exact configurations of opponents at each game. Currently, it is unclear how sensitive the results are to different four-player lineups.

  3. Appendix B.3 treats the losing offer plus ten as the minimum amount needed to win. But sometimes the winner may not be able to construct that payment from its available money cards. For example, a player whose only positive-value card is 100 must offer 100 to beat an offer of 50, offering 60 is impossible. Therefore, this metric can make unavoidable spending appear inefficient. Moreover, since the opponent’s offer is hidden, paying more than needed is not necessarily a decision error.

  4. The PDF title and the OpenReview title are different; the numbers in Fig. 2 and Fig. 4 overlap with the confidence interval bar.

References:

[1] Andon Labs. (2025). Vending-Bench Arena. https://andonlabs.com/evals/vending-bench-arena

Questions for authors
  1. Could the authors motivate what practical problems the benchmark solves that existing, more sophisticated benchmarks like Vending Bench do not?
  2. Could the authors provide the exact split of the 182 four-player configurations and details on how this schedule was decided?
1 comment

Dear reviewer rSZN,

thank you for your time and careful review, especially the banknote example and the questions about the benchmark’s contribution.

We study economic allocation with complementary assets, limited working capital and private information, as encountered in procurement and trading. A procurement agent must assemble a useful bundle without exhausting the cash needed for the remaining inputs. Kuhhandel represents this through quartet completion and portfolio-dependent scoring. Auctions create competition for assets; buy-right trades off paying to retain an asset against receiving cash for selling it. Hidden offers and zero-value cards allow bluffing about payment, while transaction histories reveal only part of opponents’ resources. Discrete payments constrain feasible deals, and donkey payouts change available liquidity. These mechanisms make a purchase affect both the portfolio and subsequent bargaining power. Vending-Bench Arena evaluates competing businesses. Cattle Trade adds a compact, established economic game with exactly checkable payments and outcomes and balanced multiplayer comparisons. We have made this economic motivation explicit in the abstract and introduction.

A constraint solver fixed all 182 lineups before play: 60 games with four LLMs, 90 with three LLMs and one heuristic, 30 with two of each, and two with one LLM and three heuristics. Each of the 14 policies played 52 games, once in every seat per deck seed across 13 seeds. Every pair met 12 times; no four-policy lineup repeated. The schedule is in the anonymous code release.

On opponent sensitivity, no remaining agent moved more than two rank positions when we excluded games involving any one participant. We repeated this check for all 14 agent configurations: remove its 52 games, recompute margin scores from the remaining 130 games, and compare the ordering of the other 13 agents with their original ordering. Thus, the ranking within this pool is not strongly driven by games involving a single opponent. This does not establish robustness to an entirely different opponent pool. Appendix B reports this check.

The earlier study also tested different opponent lineups: each of six LLMs played four games against each of seven combinations of the code agents (168 games). Appendix B reports performance for each opponent combination.

We agree that losing-offer-plus-ten can be infeasible and that a hidden opposing offer cannot define a known optimal payment. We removed that measure. Offer-card expansion measures a distinct behaviour: cards offered divided by the fewest cards in the same hand paying the same amount. A single 100 note has ratio one. Nonempty zero-value offers and empty offers are counted separately. We verified inventories for 8,455 of 8,468 positive offers across all 424 games, including every positive frontier offer (2,432). This measures visible stack construction, not wasted spending or inferred intent.

Figures 2 and 4 now separate percentages from confidence-interval bars. The titles of the paper and in Open Review are now aligned.

We hope that the clarified motivation, schedule and opponent-sensitivity results, together with the corrected analysis, address your concerns. We would be grateful if you would consider updating your assessment in light of these changes, and we remain happy to answer any further questions before the discussion closes.

Reviewer_drHyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CattleTrade, a multi-agent benchmark adapted from a tabletop card game that combines auctions, hidden-offer trading, bluffing, discrete resource management, and multiplicative scoring within a long-horizon interactive environment. The authors conduct a balanced league of 182 games involving 11 LLM policy configurations and three deterministic heuristic agents, supplemented by 242 games involving an earlier group of models. Using win rates, TrueSkill, margin scores, and behavioral logs, the paper identifies distinct trading and resource-allocation styles, observes model-dependent benefits from increased reasoning effort, and highlights opponent modeling and endgame spending as potential weaknesses of current models.

Critical strengths
  1. Auctions and buy-right decisions affect liquidity, hidden offers introduce incomplete information, discrete currency constrains payments, and multiplicative scoring makes individual transactions depend on the broader portfolio. Together, these mechanisms provide a meaningful setting for evaluating whether agents can coordinate local decisions with long-term objectives.

  2. Each policy participates in 52 games, every pair of policies meets 12 times, and seat positions are balanced within each deck seed. The authors also examine sensitivity to scoring parameters, agreement between the two league halves, bootstrap rank intervals, and seat effects. These analyses strengthen the credibility of performance comparisons within the evaluated policy population.

Critical weaknesses
  1. Table 13 assigns a negative score to bidding below an opponent’s known wealth and a positive score to offering just above it. However, knowing an opponent’s budget does not establish that exceeding that budget is strategically optimal; the decision also depends on the contested asset’s value, the opponent’s likely offer, and future uses of liquidity. Similarly, defining the endgame by having at most two cards remaining in the deck does not establish that cash has no continuation value, since subsequent trade challenges may still require funds. These metrics therefore measure adherence to particular heuristics without sufficiently establishing their strategic validity.

  2. Appendix E.1 explicitly acknowledges that the rules prompt does not clearly specify the single-card transfer case and presents an example in which a model incorrectly predicts quartet completion after a trade. This leaves open whether some failures reflect insufficient task specification rather than deficient strategic reasoning. Although Appendix E.2 describes scenario variants with clarified rules, structured inputs, and exact arithmetic assistance, quantitative comparisons across these conditions are not reported. Without such controls, attributing the observed failures to inadequate strategic integration or opponent modeling remains insufficiently supported.

  3. The non-LLM baselines consist of three hand-designed deterministic agents, without human players or stronger planning-based policies as reference points. This makes it difficult to determine whether the observed failures concern basic task understanding or genuinely challenging strategic decisions. Furthermore, the paper emphasizes the joint deployment of multiple skills but does not provide mechanism ablations, such as removing hidden offers or simplifying currency constraints, to establish how these components affect difficulty and model performance. The results demonstrate that the game differentiates policies, but provide limited direct validation of its intended measurement of integrated strategic competence.

Questions for authors

See above

1 comment

Dear reviewer drHy,

thank you for your time and careful review, particularly your questions about the behavioural measures and rule specification.

We removed the original Fig. 3, its heuristic scores, and the portfolio-versus-opponent competence claim. Knowing an opponent’s budget does not establish a good offer. Trading continued after the final auction in 161 of 182 games, with 394 subsequent trades. Final points determine the outcome; money contributes no final points. Converting cash into scoring sets therefore matters, but the last draw is too early to treat remaining cash as wasted: it can still fund trades. We now report points per outgoing coin alongside offer-card expansion, zero-value offers, cash commitment and observed transfers.

We clarified both transfer cases and tested all six Anthropic model/effort configurations from the league, plus GLM-5.3, using their published generation settings. Each policy answered the same 12 transfer questions under the original and clarified wording, and eight score-change questions under prose, JSON and JSON with precomputed pre-trade totals.

PolicyTransfer: original → clarified (out of 12)Score: prose / JSON / assisted (out of 8)
Opus 5.5 (low)12/12 → 12/128 / 8 / 8
Opus 5.5 (high)12/12 → 12/128 / 8 / 8
Fable 5.1 (low)12/12 → 12/128 / 8 / 8
Fable 5.1 (high)12/12 → 12/128 / 8 / 8
Sonnet 5 (low)6/12 → 8/124 / 5 / 7
Sonnet 5 (high)9/12 → 11/124 / 5 / 8
GLM-5.39/12 → 12/124 / 4 / 7

All 336 answers were complete and scorable. Opus and Fable were correct throughout. Clarification and precomputed accounting increased the correct-answer totals for Sonnet and GLM. This was not uniform: for Sonnet at low effort, clarification corrected four transfer answers but introduced two errors. We now distinguish these factual errors from strategic interpretations. Appendix E reports the case selection, extraction rules and results by transfer type.

Supplementary tests with Haiku 4.5 from the earlier tier gave 8/12 → 8/12 for transfers and 8/8, 7/8, 7/8 for score changes. One ambiguous rule answer per wording counted as unsuccessful. Appendix E documents its separate generation settings.

We invested substantial effort in refining one shared prompt, including auction information and response instructions; the iterations are preserved in the logs. Different formulations trade off clarity, detail and conciseness, so we selected one common prompt for the league.

We added 52 full-deck four-code-agent games: Tracker won 25, SetRace 17, RandomAgent 6 and Economy 4; all games completed. We also found that 984 of 6,870 paid frontier auctions (14.3%) required payment above the quote because change was unavailable. These automatic settlements show how often the no-change rule affected payments. Human/planning baselines and full-game mechanism ablations remain untested.

We hope that the new experiments and revised behavioural analysis address your concerns. We would be grateful if you would consider updating your assessment in light of these changes, and we remain happy to answer any further questions before the discussion closes.

Reviewer_bKqzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper turns the card game Kuhhandel into CattleTrade, a four-player benchmark with buy-right auctions, face-down trade offers that allow bluffing, money without change, and multiplicative scoring. A balanced 182-game league ranks 11 LLM policies (eight current frontier models, three at two effort levels) and three code agents, alongside an effort ablation and a behaviour profile. The authors conclude that frontier models have overtaken heuristics that beat most earlier models, that neither price nor reasoning budget predicts rank, and that models play their own portfolio well but rarely exploit opponents.

Critical strengths
  1. A balanced league with uncertainty reported. A constraint solver fixes the schedule so that every policy plays every seat once per deck seed, every pair of policies meets 12 times and no table repeats. Rankings come with bootstrap intervals, the two halves of the schedule give almost the same order (Spearman 0.92) and the paper also states that the top three policies cannot be separated. The engine validates every action and logs tokens and cost for every call. This makes the relative ranking credible at the resolution its intervals support and the design can be reused for other multi-player games.

  2. The effort ablation shows that effort labels are not comparable across models. The high-effort and low-effort runs of each model are paired by deck seed. All three models ranked higher at high effort and the gain is significant for Sonnet 5. Yet high effort multiplies hidden reasoning by about 96 for Sonnet 5 but by about 3 for Opus 5.5, although both are Anthropic models (Table 5). This supports the paper's conclusion that a model name does not fully describe an agent so evaluations should report effort settings.

Critical weaknesses
  1. The behavioural conclusions and several abstract claims go beyond the evidence. Two of the portfolio axes in Fig. 3, "complete own quartet" and "pay for cheap multiplier", get worse as models get stronger (rank correlations of about −0.75-0.75 with the Table 3 margin). Their −1-1 label (Table 13) ignores price and the chance to win the fourth card later by trade, so they may measure eagerness to buy rather than skill. Meanwhile, the axis most tied to rank, milking a needy auctioneer, targets opponents, so "blind to opponents" is too strong. The "endgame cash" axis assumes money is worthless at the end, but the paper never says what happens after the deck runs out, when cash may still matter. The abstract and conclusion also state the "overtaken" result as a finding, although §4.5 compares the two tiers only descriptively, and "neither price nor reasoning budget predicts rank" rests on a non-significant result from 11 policies, while within each model higher effort never ranked lower. Finally, the claim that long games reveal what "isolated tests do not measure" is untested, since the scenario suite (Appendix E.2) reports no results, and its premise that benchmarks test these skills "largely in isolation" overlooks AucArena (Chen et al., arXiv 2023), LLMArena (Chen et al., ACL 2024) and NegotiationArena (Bianchi et al., ICML 2024).

  2. The benchmark is not yet usable by others or well anchored. The submission is in the datasets and benchmarks area, yet the paper does not say whether the engine, code agents, verbatim league prompts, schedule or logs will be released. The Reproducibility Statement only points to their descriptions in the appendices, and there is no protocol for rating a new model. The rules never state when a game ends or what the turn limit is. The code agents are weak and underspecified reference points: §3.3 describes them in three sentences, and in 52 league games TrackerAgent completes no chicken, cat or donkey quartet (Fig. 12). With no random, search-based or human reference, the headroom of the benchmark is unknown. The evaluation is also narrow, as the Limitations partly acknowledge: one game configuration, one prompt and high effort for Anthropic models only. Finally the appendix needs cleaning, since the captions of Figs. 16 and 17 describe panels they do not show.

These two weaknesses decide my rating and I would raise it given a firm release commitment, a rescoped abstract and conclusion, behaviour axes rescored from the existing logs with the opponent-modelling claim adjusted to whatever they support, and complete rules and code-agent specifications.

Questions for authors
  1. Will you release the engine, code agents, verbatim league prompts, schedule and logs, and how would a new model be rated against the existing pool?
  2. How many decisions does each cell in Fig. 3 rest on? If the portfolio axes were scored against actual score gain, allowing for later trades, would the contrast with opponent play still hold?
  3. What happens after the deck runs out, and what is the turn limit?
  4. "Bluff Trade" on the title and the abstract vs. "Cattle Trade" in the paper?
1 comment
AuthorsOpenReview ↗

Release confirmation and answers

Dear reviewer bKqz , thank you for your time and careful, constructive feedback, and for recognizing the balanced league design and reported uncertainty. Code, data and new models. We commit to releasing the engine, code agents, verbatim league prompts, schedule and experiment data. A basic experiment runner is available anonymously; trajectories and decision logs will follow after acceptance. Appendix C now gives fuller code-agent specifications.

A new LLM joins through four-player games with established LLMs, varying opponents, seats and deck seeds under fixed settings. These opponents connect it to the existing TrueSkill pool. Retain their rating distributions, initialize the newcomer with high uncertainty, and update all participants from the final rankings, including ties. The entire league need not be rerun. The anonymous release includes an importer and the published rating prior for these updates.

Behaviour metrics and counts. We agree that completing a quartet need not justify its price, and that exceeding an opponent's budget need not be a good offer. The original axes do not establish the portfolio-versus-opponent competence contrast. We have removed both the figure and that claim; we have not established that the contrast survives the proposed rescoring. The replacement reports observed offer-card expansion, zero-value offers and cash commitment, with per-policy counts. Expansion measures the visible stack size relative to the fewest cards in the same hand paying the same amount; zero-value offers remain separate. These describe choices without labelling them strategically correct. We computed expansion across all 242 earlier and 182 frontier games. The table below gives the requested counts for every cell of the original Fig. 3; repeated bidding decisions count separately.

Game ending. After deck exhaustion, trade challenges continue. The game ends when the deck is empty, no auction is pending and nobody has a legal trade challenge; the safety cap is 200 turns. All 182 league games finished in 40–69 turns. There were 394 trades after the final auction in 161 games. Cash therefore still matters after deck exhaustion; we removed the contrary “endgame cash” interpretation.

Claims and presentation. We narrowed the abstract and conclusion to the descriptive older/frontier comparison and distinguished cross-model cost results from within-model effort effects. We removed the unsupported comparison with isolated tests and added AucArena, LLMArena and NegotiationArena. Thank you for these pointers. We added 52 full-deck games with the existing RandomAgent and the three code agents: Tracker won 25, SetRace 17, RandomAgent 6 and Economy 4; all games completed. Appendix B gives the fixed seeds, seat balance and policy specification. This adds a random reference; human/planning headroom remains unmeasured. We corrected the appendix captions and use Cattle Trade consistently in the paper; the OpenReview title will be aligned.

Decision counts for the original Fig. 3. O/F/S denote Opus 5.5/Fable 5.1/Sonnet 5; hi/lo denote effort; G3.8/G3.6 are Gemini Flash; Sol is GPT-6 Sol; GLM/DS are GLM-5.3/DeepSeek 4.1 Flash.

AxisO hiO loF hiF loS hiS loSolG3.8G3.6GLMDS
Keeps a card it needs as auctioneer A26962394028464850585149
Completes its own quartet A5340264284208247423201300301322221
Pays for a cheap multiplier quartet A6144969469115269846510416286
Broke: auctions instead of empty trade M32141911284173121151
Milks a needy auctioneer A1444543505681598597551548420561617
Pressures a completer A3101990710891014846795950874893911984
Blocks a completer as auctioneer A43333344747704052284541
Uses known opponent wealth T21481811191191001251002131248571
Spends cash in the endgame E1437335530530348466494471455364403

We hope that the anonymous code release, clarified rules and revised behavioural analysis address your concerns. We would be grateful if you would consider updating your assessment in light of these changes, and we remain happy to answer any further questions

Comments on the paper

1 comment
Area_Chair_bAtNOpenReview ↗

Reminder: Please Review the Author Rebuttal

Dear Reviewers,

The authors have recently submitted their rebuttal to your review.

As the rebuttal period is about to end, could you please take a moment to review their responses and provide a brief follow-up comment indicating whether your concerns have been addressed or if any issues remain?

If appropriate, please also consider updating your assessment based on the new information.

Thank you for your time and contributions!

Best regards, Area Chair ICLR 2027

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.