- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The work proposes Cattle Trade, a benchmark based on a four-player version of the existing trading card game Kuhhandel, combining bargaining, deception, and resource constraints. The evaluation includes eight LLMs (three tested at both low and high reasoning effort) and three rule-based agents, resulting in 14 player configurations. Each configuration plays 52 games, with 182 games in total. The paper provides quantitative win metrics as well as qualitative analysis.
Critical strengths
- The work studies frontier models like Opus 5.5, Fable 5.1 and Sol 6, which were released closer to the paper submission deadline.
- Qualitative analysis is good. The paper discusses spending, trade activity, bluffing, and specific rule-understanding mistakes.
- Implementation details provided in the appendix are useful for reproducibility.
Critical weaknesses
-
The additional value of this benchmark is not sufficiently established. Table 1 distinguishes Cattle Trade from other benchmarks through different parameters but does not highlight what practical problem the benchmark addresses. For example, Andon Labs's Vending Bench Arena [1] already evaluates frontier models on simulations involving competing economic agents.
-
Win rate depends on the exact configurations of opponents at each game. Currently, it is unclear how sensitive the results are to different four-player lineups.
-
Appendix B.3 treats the losing offer plus ten as the minimum amount needed to win. But sometimes the winner may not be able to construct that payment from its available money cards. For example, a player whose only positive-value card is 100 must offer 100 to beat an offer of 50, offering 60 is impossible. Therefore, this metric can make unavoidable spending appear inefficient. Moreover, since the opponent’s offer is hidden, paying more than needed is not necessarily a decision error.
-
The PDF title and the OpenReview title are different; the numbers in Fig. 2 and Fig. 4 overlap with the confidence interval bar.
References:
[1] Andon Labs. (2025). Vending-Bench Arena. https://andonlabs.com/evals/vending-bench-arena
Questions for authors
- Could the authors motivate what practical problems the benchmark solves that existing, more sophisticated benchmarks like Vending Bench do not?
- Could the authors provide the exact split of the 182 four-player configurations and details on how this schedule was decided?
Dear reviewer rSZN,
thank you for your time and careful review, especially the banknote example and the questions about the benchmark’s contribution.
We study economic allocation with complementary assets, limited working capital and private information, as encountered in procurement and trading. A procurement agent must assemble a useful bundle without exhausting the cash needed for the remaining inputs. Kuhhandel represents this through quartet completion and portfolio-dependent scoring. Auctions create competition for assets; buy-right trades off paying to retain an asset against receiving cash for selling it. Hidden offers and zero-value cards allow bluffing about payment, while transaction histories reveal only part of opponents’ resources. Discrete payments constrain feasible deals, and donkey payouts change available liquidity. These mechanisms make a purchase affect both the portfolio and subsequent bargaining power. Vending-Bench Arena evaluates competing businesses. Cattle Trade adds a compact, established economic game with exactly checkable payments and outcomes and balanced multiplayer comparisons. We have made this economic motivation explicit in the abstract and introduction.
A constraint solver fixed all 182 lineups before play: 60 games with four LLMs, 90 with three LLMs and one heuristic, 30 with two of each, and two with one LLM and three heuristics. Each of the 14 policies played 52 games, once in every seat per deck seed across 13 seeds. Every pair met 12 times; no four-policy lineup repeated. The schedule is in the anonymous code release.
On opponent sensitivity, no remaining agent moved more than two rank positions when we excluded games involving any one participant. We repeated this check for all 14 agent configurations: remove its 52 games, recompute margin scores from the remaining 130 games, and compare the ordering of the other 13 agents with their original ordering. Thus, the ranking within this pool is not strongly driven by games involving a single opponent. This does not establish robustness to an entirely different opponent pool. Appendix B reports this check.
The earlier study also tested different opponent lineups: each of six LLMs played four games against each of seven combinations of the code agents (168 games). Appendix B reports performance for each opponent combination.
We agree that losing-offer-plus-ten can be infeasible and that a hidden opposing offer cannot define a known optimal payment. We removed that measure. Offer-card expansion measures a distinct behaviour: cards offered divided by the fewest cards in the same hand paying the same amount. A single 100 note has ratio one. Nonempty zero-value offers and empty offers are counted separately. We verified inventories for 8,455 of 8,468 positive offers across all 424 games, including every positive frontier offer (2,432). This measures visible stack construction, not wasted spending or inferred intent.
Figures 2 and 4 now separate percentages from confidence-interval bars. The titles of the paper and in Open Review are now aligned.
We hope that the clarified motivation, schedule and opponent-sensitivity results, together with the corrected analysis, address your concerns. We would be grateful if you would consider updating your assessment in light of these changes, and we remain happy to answer any further questions before the discussion closes.