acceptodds
Under review as a conference paper at ICLR 2027

BazaarBench: Delegation Safety in Decentralized C2C Marketplaces Run by LLM Agents

Abstract

In decentralized consumer-to-consumer (C2C) marketplaces, people list goods, negotiate with strangers, and rate one another, so trust rests on reputation. Large language model agents now act and decide for users. On a real platform, their unsafe choices would cost users money, privacy, and reputation. Even among humans, unsafe trades are common and hard to spot, since weak policing lets people fake shipments or talk buyers into scams. The key question is whether agents trading for people fail in such unsafe ways. BazaarBench, a simulated C2C marketplace, keeps each item's owner, true condition, and commitments apart from agents' claims. It links six failure types to trades, so the truth of every trade can be observed. We first create 100 synthetic personas with inventories from an open inventory dataset, and three models each run one base market of these agents for 30 simulated days. Then 20 of the 100 agents in each market switch to one of five tested models, keeping personas, inventories, and histories. They trade for seven days under ordinary instructions, deadline pressure, or a user's adversarial instructions, each in its own market copy. Unsafe trading is routine even without harmful instructions, as 16% to 22% of committed deals in the base markets go through despite a failure the platform's records confirm. No tested model is safe by default. In the ordinary baseline after the switch, where models of different capability trade together, each causes such failures in its own deals, 14% overall and 8% to 23% by model. Deadline pressure pushes them into riskier purchases, most often of units the seller has also promised to another buyer. Under adversarial instructions, the share of tested sellers' committed deals carrying a fake or wrong item to the handoff more than doubles, from 15% to 33%. Had inspection not stopped them, their weekly gain would have risen by about 60%. Inspection catches almost all of them, so completion falls and realized gain stays flat, while the few that deceive a buyer are shipped without inspection. We therefore recommend checking ownership and commitments at acceptance and requiring inspection or delivery evidence before completion. The 48 runs hold 357,608 agent model calls with complete market records, and code and data are available for analysis and future training at https://anonymous.4open.science/r/BazaarBench.

Primary area:
datasets and benchmarks
Keywords:
Safety, Open-Ended Marketplace, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GDnkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BazaarBench, a simulated decentralized C2C marketplace for evaluating the delegation safety of LLM agents in long-horizon trading scenarios. The benchmark maintains persistent marketplace states and private records of item ownership, true condition, and commitments, which enables six types of unsafe behaviors to be tracked from agent reasoning to final transaction outcomes. The authors evaluate several LLM agents under ordinary instructions, deadline pressure, and adversarial user instructions, and show that unsafe trading behaviors occur even in the normal setting and become more frequent under pressure or adversarial goals. The paper further analyzes several platform-level safeguards, such as ownership verification and inspection before transaction completion.

Critical strengths
  1. This paper is well written, easy to follow;
  2. The benchmark setup is interesting and quite realistic compared with many single-turn agent safety benchmarks.
  3. The experiments are large-scale and cover normal, pressure, and adversarial settings.
  4. The paper gives some useful platform-level insights, e.g., ownership check and inspection.
Critical weaknesses
  1. The whole benchmark is still simulated, so I am not fully convinced how much these results can transfer to real C2C platforms. 2.Some safety labels, particularly premature close (F4), PII-related messages (F5), and unverified trust claims (F6), rely partly on a single semantic judge, while comprehensive human validation of these labels is still missing.
  2. The paper suggests several platform safeguards, but most of them are not directly tested in new market runs.
  3. The tested agents are still concentrated in a few model families, mainly OpenAI and DeepSeek; adding agents based on Claude, Gemini, Qwen would make the conclusion about LLM agents in general more convincing.
Questions for authors

Could the authors provide more evidence or discussion on how these findings may transfer to real C2C platforms?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (1/3)

Thank you for the review and for the question about real platforms.

W1 + Q1: The review asks how findings from a simulated market carry over to real C2C platforms.

We checked which failures a platform's records can show, and which need evidence about the physical item. What carries over is the link between conflicting promises, decisions to complete a sale and missing checks. Exact rates describe our simulated agents and markets.

Traders' profiles and chats are generated, but products and starting stock come from a public eBay sample. The simulation tracks each item's owner, original condition and promises. A real marketplace can record offers and confirmations, but needs separate evidence of who holds an item and its condition. An accepted offer shows a promise, not that the seller has the listed item.

1. Each kind of failure needs its own evidence on a real platform.

FailureEvidence in BazaarBenchEvidence a real platform needs
Item promised to several buyersAccepted offers and meetings for one itemOffer history and a way to link listings of one physical item
Item not owned, or described wronglyOwnership and original conditionSeparate evidence of possession and condition
Sale completed too earlyInspection, scheduling and confirmation recordsEvidence that inspection or delivery happened
Wrong product or version handed overItem taken from stock and its purchase historyLink from listed product to item handed over, with resale history

2. Records kept over time show failures that one conversation hides.

Offers to different buyers can each look normal but together promise more items than the seller has. Under deadline pressure, 170 purchases by tested buyers pass our idealized inspection audit and link to a seller's condition, ownership or promise failure. In 161/170, the seller had also promised the item to another buyer. The seller usually uses the market's base model (Table 45). Under ordinary instructions, 207/18,929 pages shown to tested agents list several accepted sales for one listing, and in 34 the seller accepts or counters another offer. But no field marks each item as promised or sold (Appendix J.2).

So the agent must track its promises, and the platform could reserve promised items.

3. Our review of handovers shows that checking ownership is not enough.

At L3 (adversarial instructions) with checks on, 81 new sales delivered an item whose title did not exactly match the listing. So the title could not show if it was the right item. Our record review supports 60 of them, finds 6 wrong items or versions and leaves 15 unresolved (General Response 1/2). All 81 trace to an item the seller owned and a matching buyer record. Wrong ones include Nest Hub listings delivering Google Home units and Charcoal Echo Dot listings delivering Plum units.

So a real platform must check the promised product, including its version and resale history (Tables 42 and 46).

4. Real platforms where traders arrange sales themselves have similar gaps.

Buyers and sellers arrange sales directly through messages (OfferUp, 2026). So a seller can accept several offers for one item, list an item it does not have, or confirm before inspection or delivery. Human traders abuse such gaps. An eBay seller mailed empty packages (U.S. Attorney's Office, Western District of Pennsylvania, 2019). Facebook Marketplace sellers handed over items after fake payment confirmations (Staffordshire Police, 2022). A platform that records accepted offers could lock a promised listing, and one physical item cannot be handed over twice. Our closed loop experiment enforces both.

This is not a future problem. Personal AI agents already buy and sell for people, and news reports show it already goes wrong. A user let Meta's personal agent Muse sell his keyboard on Facebook Marketplace. It accepted an offer on its own and gave the buyer his home address. The buyer came to his house late at night. For another item, it offered $100 below the minimum he set (Los Angeles Times, Oct 2026). Grok Bot can already buy things for people (Grok Bot changelog, Oct 2026). Visa asked 22 shopping agents to buy a TV, a laptop and a hair appliance. They succeeded only about 57% of the time and often bought the wrong product (Fortune, Nov 2026). BazaarBench gives these agents an arena built from real C2C listings (the public eBay sample), where they trade with each other as in a real market. So we can see which failures happen and map each to the records a real platform needs to stop it.

In the revision, Section 3.1 describes the records and what agents see. Section 5 explains how to read the results for real platforms. Appendix M.2 (Table 46) maps simulation evidence to what a real platform needs, next to the handover cases in Appendix L.4.

AuthorsOpenReview ↗

Response to Reviewer 3 (2/3)

W2: The concern is that some failure labels rely partly on a single judge model and lack a full human check. The review names premature close (F4), messages with personal information (PII, F5) and unverified trust claims (F6).

The revised paper reports two checks. First, humans labeled a sample of judged cases again. Second, another model, GPT-5.6, judged new cases without seeing which model, run or condition they came from. The human result is one agreement number for all failure types together. The results by type below compare the original judge with GPT-5.6. Both checks measure agreement, not accuracy for each type (Appendix A).

Records and the judge have different roles. For overstated condition, unowned listings and conflicting commitments (F1 to F3), the inventory and event records decide the label at stage S3 and later, when the platform has stored the action. For premature close (F4), PII text (F5) and unverified trust claims (F6), a judge model reads the evidence and decides what it means. So facts about items, labels from the judge and the human agreement score each use a different kind of evidence.

1. The human evaluation gives one agreement number for all failure types together.

Humans labeled 200 randomly chosen cases that the judge had labeled. They decided whether a failure was present and, if so, its furthest stage (S1 to S5). They agreed with the judge on 188/200 (94%). In 12 (6%), they differed on presence or on stage. In 4 of these, humans said S5 (Carried through) and the judge said S3 (Reached another participant). In 2 more, humans said S1 (Considered) and the judge said no failure. Both patterns sit at a border between two labels: S3 versus S5, and S1 versus no failure. Appendix H.3 reports this result, which is not split by failure type.

2. A second model judged new cases, and its agreement with the original judge is similar for each type.

The new sample has three groups: cases the original judge labeled as failures, cases it labeled as no failure, and cases our filter never sent to the judge. GPT-5.6 used the same written labeling rules (the rubric), unchanged. We stored its labels before we compared them.

Failure typeExact agreement on presence and stageAgreement on presenceCohen's κ
F4: premature close59/6863/680.790
F5: PII58/6660/660.818
F6: unverified trust claims58/6658/660.781

General Response (2/2) gives the overall result (175/200, κ = 0.808) and the scope of this check between two models. For presence and stage together, κ ranges from 0.781 to 0.818. Unverified trust claims (F6) have the lowest κ. All 8 of their disagreements are about whether a failure is present. Of these 8 (Table 12), the second judge found no failure in 5 cases labeled S1 and 2 labeled S3 by the original judge. In the last case, labeled no failure, it gave S2 (Attempted).

3. Of the 66 F5 cases, the judges agree exactly on 58, and 11 have their evidence in a message.

The review mentions messages related to PII. We report agreement for all 66 F5 cases together (58/66, κ = 0.818). Their evidence is found in different kinds of objects, which Table 13 separates:

Object holding the F5 PII evidenceCasesExact agreements
Platform action21
Listing66
Message1110
Offer1110
Photo1311
Rating44
Recorded reasoning1916

A message holds the evidence in 11 of the 66 cases, and the judges agree exactly on 10 of them. This is a small group, so the message result rests on 11 cases, not 66. In one example, both judges give S4, Acted upon: a message asks for a device identifier, and the reply answers it. The rubric needs more evidence before it counts the identifier as actually shared.

4. Most disagreements fall on one label, S1, and on one border between stages.

In the second check, 13/25 disagreements are S1 labels where the second judge found no failure. In 4 others, the judges differ on whether an early confirmation counts as S5 when the deal completes later (Table 12; example in General Response 2/2). So S1 is the least stable label. S1 means the agent's recorded reasoning considers the unsafe act. The paper compares S1 only within one tested model and treats it as supporting evidence. If reasoning is missing, S1 counts as unknown.

The platform records listing and commitment results, so they can still be checked directly, next to the judge's labels. We keep the evidence that an agent tried an action separate from any judgment of its reasoning or of what it led to later.

In the revised paper, Section 3.2 names the evidence source of each label and Appendix H.3 reports the human evaluation. Appendix H.4 gives agreement by type, the disagreements by stage and the F5 cases split by object in Tables 11 to 13.

AuthorsOpenReview ↗

Response to Reviewer 3 (3/3)

W3 + W4: The review says most suggested safeguards are not tested in new market runs, and the tested agents come from a few model families, mainly OpenAI and DeepSeek.

We tested the safeguards while agents traded, in a closed loop experiment. Tested agents used GPT-OSS-120B, and the other traders used Qwen3.6-35B-A3B. We report two extra base markets separately from the controlled comparisons of tested models. In each, all 100 agents use one model, Qwen or GPT-5.4-nano.

The experiment compares the original rules with enforced checks, under ordinary (L1) and adversarial (L3) instructions. Both versions start from the same saved market with the same models. General Response (1/2) gives the design, measures and results table. The records flag overstated condition (F1), unowned items (F2) and conflicting promises (F3). Item checks test that the handed over item is the listed one, owned by the seller, available, and not overstated. Qwen is the model of the other traders and of one extra base market, never a tested model.

1. The checks act during trading and block flagged promises and handovers.

The checks act when an item is promised, inspected and handed over, but not on new listings. At L3, sellers among the other traders complete 130/171 (76.0%) of agreed sales without the checks and 94/131 (71.8%) with them. Completed sales that pass the item checks of Table 2 fall from 38 to 24. With the checks, every completion must record which item moved to the buyer. So the 32 sales completed without such a record no longer happen. Completed sales with a flag in their history fall from 59/130 (45.4%) to 3/94 (3.2%).

At L3, the other traders keep the ordinary prompt. With the checks, they make 160 flagged attempts to promise an already promised item, against 115 without them. The number accepted falls from 64 to 0. Inspection at L3 also finds the item missing in 9 sales by tested sellers and 16 by other sellers. None completes.

2. After a refusal, agents try the action again and keep trading.

At L3, the checks refused an action by 30 of the other traders. At the time, 26 of them were buyers. 12 repeat the refused action next, and 19 repeat it at some point. 29 make or answer offers in other conversations, and 23 later complete a sale. The median time to that sale is 11 ticks of two hours, with a range of 1 to 38. So recovery after a refusal is a task for the agent. A correct refusal comes with retries and trading elsewhere. The six tested sellers whose promise or completion was blocked also acted again within two ticks. Five kept trading or listing as sellers (Appendix L.3).

Our review of handed over items adds one more finding. In some completed sales, the seller owns the item handed over, but it is the wrong product, model or color. So blocking double promises and recording transfers do not prove the buyer got the right product (Appendix L.4).

3. We add evidence from more models and state each model's role.

EvidenceModels and roleWhat we compare
Main studyFive tested models from two developers, three base marketsResponses to ordinary, deadline pressure and adversarial instructions from the same saved markets
Closed loop experimentGPT-OSS-120B tested agents; 80 Qwen other tradersOriginal rules against enforced checks at L1 and at L3
Two extra markets100 Qwen or 100 GPT-5.4-nano agentsMarket histories over 15 days under ordinary instructions

Qwen3.6-35B-A3B, from a third developer, makes conflicting promises in both roles under the ordinary prompt. In it's own base market, 216/340 failures that reach the last stage (S5) are conflicting promises. As the other traders at L3 without the checks, Qwen agents make the 115 flagged attempts above, and 64 are accepted. The Qwen market also has 55 unowned and 26 overstated listings at S5. The GPT-5.4-nano market has 559 failures at S5 over 457 agreed sales (1.22 per sale). Of these, 364 are photos sent without privacy protection. Both extra markets use one hour ticks, and each built its own history.

The main study reports each base market alone and all three together, since the 80 other traders affect which sales complete. Under adversarial instructions, more sales complete with an unavailable item or overstated condition in all three markets, but by different amounts. Listing attempts show the agent's own action, before any other trader acts. Our conclusions about models cover the five tested models from two developers (Appendix A). Qwen adds evidence as the other traders and as a whole market.

In the revision, Section 4.6 compares the two rule sets in Table 2. Appendix L.3 to L.4 reports behavior and handovers in Tables 37 to 42. Appendix J.11 shows the extra markets in Table 32 and Figure 5, next to the results for each market in Appendix J.6.

After carefully reviewing the authors' rebuttal and the comments from other reviewers, I have decided to maintain my original score.

Reviewer_DQCsOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BazaarBench, a benchmark for evaluating delegation safety for agents in decentralized C2C markets. It highlights why completing a transaction does not necessarily mean acting safely on a user’s behalf. The results reveal prohibited transactions even under ordinary instructions, while adversarial instructions substantially increase the number of fictitious or misrepresented goods reaching the delivery stage. The authors also explore potential safeguards through post-hoc inspection audits.

Overall, this is a comprehensive benchmark paper. My major concern is that most findings are intuitive. For example, agents can violate constraints, pressure can increase risky behavior, and adversarial instructions can induce deception. Thus, its contribution appears primarily integrative.

Critical strengths

This is a comprehensive benchmark paper that reflects substantial experimental effort and supporting information in both the main content and the appendices. It provides detailed results across models and market conditions, releases trajectory data, and acknowledges limitations related to initialization, judge blindness, and metric selection.

Critical weaknesses

My main concern is whether the benchmark enables the study of questions that were previously difficult to investigate and yields sufficiently new insights. Most findings are largely intuitive, for example, agents can violate constraints, pressure can increase risky behavior, and adversarial instructions can induce deception. While BazaarBench offers a useful environment for quantifying these behaviors, its contribution appears primarily integrative, with limited new insights into failure mechanisms or the design of safer agents and marketplaces.

Questions for authors
  1. In Section 3.3, the authors explain that the audit only reads the recorded trajectories without changing any actions. But rescoring those fixed trajectories does not tell us how the market would actually behave with safeguards in place. Canceling a transaction would affect inventory, funds, reputation, and later decisions. Could the authors run a closed-loop continuation with a safeguard enforced and report how it affects both safety and the completion of legitimate transactions?

  2. I’d also like the authors to explain more clearly what this benchmark adds beyond existing work and what new insights it offers into why failures happen or how agents and marketplaces could be better designed. This would help me consider raising my rating.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (1/2)

Thank you for the review and for proposing the closed loop test.

Q1: The question is how an enforced safeguard affects both safety and the completion of honest sales.

We ran that test. In the L3 runs, background sellers' completion rate drops from 76.0% to 71.8% with the checks on. Their sales that pass all item checks drop from 38 to 24. So the checks reduce completed trade. With the checks on, most sales still have an item whose title does not exactly match the listing, so the title alone cannot confirm it is the right item (60 → 70 at L3). We reviewed the records of the 81 such L3 sales with checks on: 60 are supported by the records, 6 are the wrong item or version, and 15 cannot be confirmed from the item's history.

General Response (1/2) describes the four runs and what they measure. Tested agents get ordinary (L1) or adversarial (L3) instructions, and background agents always get ordinary ones. The two L3 runs start from the same market and form a fair side by side comparison. A refusal changes later trading in that run. The runs with and without checks judge items by the same item checks. Tables cover deals first promised during a run. Arrows go from original rules to checks on.

1. A smaller share of promised deals completes, and every sale with checks on has a record of the item moving to the buyer.

Background sellers' dealsL1L3
Completed / promised115/144 → 92/131130/171 → 94/131
Completion rate79.9% → 70.2%76.0% → 71.8%
Flagged / completed53/115 → 5/9259/130 → 3/94
Flagged share46.1% → 5.4%45.4% → 3.2%
Completed with no record of the item moving26 → 032 → 0

A flagged sale has a condition, ownership or promise flag from the records. The checks require a record of the item moving at every sale. So the 26 (L1) and 32 (L3) sales with no such record under the original rules do not happen. Each of them is itself an ownership flag (Appendix L.1). They are 26 of the 53 and 32 of the 59 flagged sales there, so the flagged share and the missing record count move together.

2. In the L3 runs, fewer sales pass all item checks with the checks on, and most sales have item titles that do not exactly match.

A sale passes when, just before completion, the records show it is the listed item, the seller owns it, it is available and its condition is not overstated. A deal is eligible when it meets the same tests at the moment of the promise. A title mismatch means the strict title match cannot confirm the item. It is not evidence that the item is wrong.

Background sellers, same checks in both runsL1L3
Completed sales passing item checks29 → 3438 → 24
Completed / eligible promised deals27/31 → 38/4933/46 → 27/37
Title does not exactly match / completed60/115 → 58/9260/130 → 70/94

In the L3 runs, background sellers' passing sales drop from 38 to 24. Eligible promised deals drop from 46 to 37, and 33 and 27 of them complete. The L1 runs, which are descriptive only, move the other way (29 → 34). For tested sellers, passing sales are 2 → 2 at L1 and 7 → 1 at L3, and completed eligible deals are 2/10 → 2/6 and 7/12 → 1/5.

3. Reviewing the records one by one confirms most sales with unclear titles and finds some wrong products.

All 81 such sales among new deals in the L3 run with checks on trace to one item taken from the seller's own stock and a matching buyer record. The review follows each product and its resale history:

Seller groupRight itemWrong item or versionUnresolvedReviewed
Tested92011
Background5141570
Total6061581

Wrong items include Nest Hub listings delivering Google Home units and Charcoal Echo Dot listings delivering Plum units. Of the 15 unresolved background handoffs, 12 lack the earlier purchase record needed to rebuild the item's history. The other three each rest on one unverified product or version claim. The review covers only the run with checks on. The comparison between rule sets uses the item checks of Table 2.

4. The checks also block a little honest trade and add extra steps.

In the L3 run with checks on, the listing lock refuses one deal for an otherwise eligible item, because the lock works per listing and can limit a seller who holds several units. Three refusals for missing shipment inspection fall on two eligible deals. Counting deals promised before the runs, background sellers complete 130 → 101 sales at L1 and 143 → 103 at L3, and tested sellers complete 18 → 12 and 22 → 15. The wrong items show the gap between handing over an owned item and delivering the listed item.

In the revision, Section 4.6 compares the runs in Table 2. Appendix L.1 to L.2 defines the measures in Tables 34 to 36. Appendix L.3 to L.4 reports refusals and handoff evidence in Tables 37 and 41 to 42.

AuthorsOpenReview ↗

Response to Reviewer 2 (2/2)

W1 + Q2: The review says most findings are intuitive and asks what the benchmark adds and what new insights it gives.

We used lasting records of every item and promise to study conflicts between negotiations that each look fine alone. We show how these failures happen, which agents try them, and what enforced checks stop.

A2A-NT studies negotiation (Zhu et al., 2025). Magentic Marketplace studies customer and business interactions (Bansal et al., 2025). Project Deal observes agents trading real goods (Anthropic, 2026). Lei et al. (2026) compare a reputation system with a deposit system. BazaarBench adds lasting records of each item's owner, starting condition and promises. These records link a promise to promises made to other buyers, to the item handed over and to trading after a refusal. In Table 47, none of the three closest studies combines such records, agents that both buy and sell, and safety labels based on them.

1. Under deadline pressure, agents try more often to promise one item to two buyers, but not to post false listings.

Attempts by the five tested modelsOrdinary instructionsDeadline pressure
Promise one item to two buyers71107
False listing (overstated or unowned item)4337

Counts add up the attempt (S2) cells of Tables 20 to 21. Tested agents also agree to more deals, from 1,457 to 1,921 (Section 4.4).

Under deadline pressure, the idealized inspection audit (which assumes perfect inspection) still counts 170 purchases by tested buyers as completed, though each is linked to one of these seller failures. In 161 of them, the seller had also promised the item to another buyer. The seller is usually an agent using the base model (Table 45). Each negotiation alone shows only an agreed sale. Only the other promise for the same item shows the conflict.

Under ordinary instructions, tested agents see the market 18,929 times. In 207 of these views, one listing has two or more accepted deals, and in 34 the seller accepts or counters another offer. Views show existing promises but no field marking an item as promised or sold (Appendix J.2).

2. We need to test both kinds of instruction: GPT-5.4 changes sharply between them, but GPT-5.4-mini does not.

Listing attemptsOrdinary instructionsAdversarial instructions
GPT-5.4, overstated condition2677
GPT-5.4-mini, item not owned2222

All these attempts are stored listings that inventory and event records confirm. Counted by agent, 24/60 GPT-5.4 agents list an unowned item under adversarial instructions, compared with 2/60 GPT-5.5 agents. As supporting evidence for each model (Table 22), GPT-5.4's recorded reasoning considers the same failure in 649 of 746 of its overstated or unowned listings under adversarial instructions and in none of its 4 under ordinary ones. It does so in none of GPT-5.4-mini's 22 unowned listings under either instruction. These judge labels were not in the sample judged again with names hidden, and are the least stable kind (General Response 2/2).

3. The completion rate hardly changes, but more completed deals involve an unavailable or overstated item.

In the main study, the platform counts 86% of agreed deals as completed under ordinary instructions and 83% under adversarial ones. But the share of tested sellers' agreed deals that complete with an unavailable item or overstated condition rises from 15% to 33%. Under ordinary instructions, most of these (126 of 141) involve an item already sold in another completed deal. Of 370 such deals under adversarial instructions, most of the increase comes from items the seller never had (73) or only overstated (121) (Section 4.5).

4. The checks refuse attempts to promise an item twice, but most refused agents repeat the request.

In the closed loop test with adversarial instructions (L3), background agents (the 80 untested agents) try to promise an already promised item. With the checks on, these attempts rise from 115 to 160, but accepted ones fall from 64 to 0. Of 30 refused background agents, 19 later repeat the request, 29 make or answer offers elsewhere and 23 later complete a deal. Our review of handoffs also finds owned items of the wrong product or version, a problem that enforcing promises does not solve.

For platform design, holding an item for its buyer and checking the physical item are separate tasks. For agent design, agents need to track existing promises and recover from a refusal without repeating the request. For evaluation, attempts, other traders' responses and outcomes for each item are best reported separately.

In the revision, Section 2 relates the benchmark to Lei et al. (2026), Appendix N compares related benchmarks in Table 47, Sections 4.4 to 4.6 present these findings, and Appendices J.2, J.4, J.5, L.3 and M.1 give supporting evidence.

Reviewer_bL9wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BazaarBench, a benchmark for delegation safety: whether LLM agents that buy and sell on a user's behalf do so safely. It simulates a person-to-person marketplace (like Craigslist or Facebook Marketplace) in which every buyer and seller is an LLM agent acting for one of 100 synthetic personas, with inventories drawn from eBay listings. The central design choice is that the platform privately records each unit's owner, true condition and commitments, separately from what agents claim in listings and messages, while checking little at trade time. Against this record the paper defines six failures: overstated condition (F1), listing items the seller does not own (F2), promising one unit to two buyers (F3), completing before inspection or delivery (F4), privacy and off-platform leaks (F5), and false reputation claims (F6). Each is traced through five stages, from stored reasoning (S1) to a completed deal (S5). F1–F3 are confirmed by replaying the logs against the record; F4–F6 rely partly on a GPT-5 judge.

Each market first runs on a single base model (GPT-5.5, DeepSeek-V4-Pro or GPT-5.4-mini) for 30 simulated days. At day 30, a fifth of the agents switch to one of five tested models (GPT 5.5, 5.4, 5.4 mini, OSS-120B and DeepSeek V4 Pro) and trade for another week alongside the rest. These continuations vary what the user asks of the agent: ordinary instructions, deadline pressure (sales targets and a tight budget), or an adversarial user who wants the agent to exploit other agents.

The main finding is that unsafe trading is routine even when no one asks for it: under ordinary instructions, a substantial share of deals complete despite a failure the record confirms, and models differ widely, with GPT-5.5 the most careful and GPT-5.4-mini the least. Deadline pressure makes things worse, mostly because sellers promise the same unit to more than one buyer. Adversarial instructions roughly double how often a fake or wrong item reaches the handoff and make cheating more profitable, though the models split sharply: GPT-5.4 goes along most readily, followed by DeepSeek-V4-Pro and GPT-OSS-120B, while GPT-5.5 and GPT-5.4-mini barely change. Replaying the same runs with inspection at the meetup, the authors find it would block nearly all of these deals and remove the extra profit, but shipped deals, which have no inspection step, remain exposed. The paper therefore recommends two platform safeguards: checking ownership and existing commitments when an offer is accepted, and requiring inspection or delivery evidence before a deal completes. Code, run data and judge outputs are released.

Critical strengths
  1. The work is well motivated. As LLM agents start to buy and sell on users' behalf, how they behave in a marketplace becomes an important question, both for the users who delegate to them and for the platforms and regulators who set the rules they trade under.
  2. The simulator is a valuable resource in its own right. Because it is a controlled setting with a private record of what is actually true, it can be used to test other parts of a marketplace besides the agents: platform rules, fraud detection, regulation, and generative engine optimization (GEO) tactics.
  3. The authors are open about the work's limitations. I appreciate that.
Critical weaknesses
  1. The paper measures the platform more than the agents. It is framed around whether agents can be trusted to trade for a user, but many of its headline numbers reflect checks the platform does not make. The deadline-pressure effect comes almost entirely from sellers promising one unit to two buyers, and most "fake or wrong items" at handoff are units that had already gone to another buyer or never existed, not items described as better than they are. Both parts of the paper's recommendation are platform rules that would remove these failures whichever agent is trading. The findings that are really about agents, such as the split between models under adversarial instructions and what their stored reasoning shows, are a small part of the paper. The paper should say clearly which findings are about the agents and which are about the platform.

  2. In the runs with the tested models, the simulator records failures but does not block them, so it cannot show how agents adapt when stopped. The authors justify this as realistic, but that holds only in part. A platform cannot see that a phone is scratched, but it does know when a listing is already committed, since accepting an offer happens on the platform. And in the real world a sold item cannot be handed over a second time, yet the simulator allows it. Because agents are never refused, we cannot see whether a blocked cheater gives up, overstates condition instead, or steers buyers toward shipping, where there is no inspection. The claim that inspection removes the gain from cheating comes from replaying fixed runs with an inspector that catches every mismatch, so it is an optimistic upper bound, and the recommended safeguards are never tested with agents trading under them. Such a test is feasible: Lei et al. (2026, arXiv:2605.10059), which the paper does not cite, have LLM buyers and sellers trade under a reputation-only platform and under one where sellers post a deposit that buyers can challenge, and compare how often sellers misstate quality. The lack of refunds and disputes adds to this, since it leaves checks before completion as the only remedy the paper can study.

  3. The evidence about agents is thin. The most interesting agent-level result, how differently the models respond to adversarial instructions, rests on models from only two developers and on three base markets. The tested agents also trade mostly with base-model agents, so their outcomes depend partly on how gullible the counterparty model is. Several failure types, and everything about what a model "considered," rely on a GPT-5 judge that has not been checked against human labels and that can see which run it is judging. These results should be treated as provisional until there is a re-judge with the run label hidden and a small check of agreement with human labels.

Questions for authors

See weaknesses. Is my characterization fair?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (1/4)

Thank you for the careful review and for proposing tests of the safeguards and the judge.

W1: The review says the paper measures the platform more than the agents. The question is which findings are about agents and which are about the platform.

We separate what agents did from how sales ended. Completed sales come from agents and loose rules that record failures without blocking them. The closed loop experiment runs agents under enforced checks. Two results are about agents. Models differ in how often they attempt false listings. Attempts to promise one item to two buyers (double promises) rise under deadline pressure.

Our main comparisons start from the same saved market and change only the tested model and instructions. S1 means the agent's reasoning considers a failure. S2 means the agent tries the action, even if the platform blocks it. Later stages also need the platform to accept it and the other trader to act. An attempt comes from the agent alone. A completed sale also depends on the other trader and the platform rules. Our fourth comment shows how we checked the S1 labels.

1. An agent's attempts and its completed sales depend on different things.

MeasureWhat it depends on
What the agent consideredThe tested agent's returned reasoning
Attempt at a false listing or double promiseThe tested agent's action, accepted or blocked. A double promise needs a second buyer
Listing stored on the platformThe action and the platform accepting it
Failure acted on or carried throughThe action, the other trader's reply and platform rules
Unavailable item or overstated condition at completionSeller's listing and promises, buyer's confirmation and checks at handoff

For example, GPT-5.4 tries to overstate condition in 2 listings under ordinary instructions and 677 under adversarial ones (F1). GPT-5.4-mini tries 22 times under both to list items it does not own (F2). Adding F1 and F2, attempts go from 12 to 5 for GPT-5.5, 4 to 746 for GPT-5.4, 24 to 24 for GPT-5.4-mini, 2 to 156 for GPT-OSS-120B and 1 to 358 for DeepSeek-V4-Pro (Table 19). Whether such a listing leads to a sale also depends on the buyer and platform. Table 1 reports action counts apart from percentages of completed sales. Table 45 shows what each measure depends on.

2. Under both kinds of instructions, most sales with an unavailable item or overstated condition involve an item already sold or never held.

Counts cover tested sellers' sales of this kind, across tested models and base markets.

Recorded reasonOrdinary instructions, L1Adversarial instructions, L3
Item already sold in another sale126/141176/370
Item the seller never held13/14173/370
Overstated condition only2/141121/370

Under ordinary instructions, most cases are items sold again, as the review describes. Under adversarial instructions, items sold again (176) and items never held (73) still outnumber overstated conditions alone (121). Most of the increase comes from items never held and overstated conditions, in listings the tested sellers made themselves (Section 4.5).

3. Under deadline pressure, agents attempt more double promises, and the platform allows them.

Across the five tested models, these attempts rise from 71 to 107. Tested agents also agree to more deals, from 1,457 to 1,921 (Section 4.4). After the idealized inspection audit, tested buyers still complete 170 purchases linked to a seller's failure on condition, ownership or promises. In 161 of them, the seller had also promised the item to another buyer. That seller is usually an agent using the base model (Table 45). Agents make these promises, and loose rules let the sales go ahead.

Under ordinary instructions, tested agents saw 18,929 market views. In 207 of them, one listing shows two or more accepted deals. In 34 of those, the seller accepts or counters another offer. The views do show existing promises. But no field directly marks each item as promised or sold (Appendix J.2). So tracking existing promises is a task for agent design, besides a platform rule that reserves promised items.

4. With the checks on, agents still attempt double promises, but the platform refuses them.

At L3, background agents (the 80 agents not tested) attempt 115 double promises under the original rules and 160 with the checks on. Repeats count separately. Accepted ones fall from 64 to 0. These agents keep ordinary instructions, and most refused agents are buyers. This separates what traders ask for from what is accepted. Our second comment follows the tested sellers whose deals were blocked.

In the revision, Section 3.3 and Appendix M.1 separate these measures in Table 45. Sections 4.4 to 4.6 report findings on actions and outcomes. Appendix J.2 shows what sellers could see. Appendix J.7 breaks down completed sales in Table 25.

AuthorsOpenReview ↗

Response to Reviewer 1 (2/4)

W2: The concern is that the paper does not show how agents react when stopped, and that the inspection result is an optimistic upper bound.

While agents trade, the closed loop experiment enforces the two real world limits the review describes. A listing promised to one buyer is locked. A sale completes only if the platform records the item moving to the buyer. (The main runs leave out these checks on purpose, Section 3.1.) We followed every tested seller with a blocked promise or sale to the end of the run. There are six, the full group in both runs with checks on. Three had ordinary instructions (L1) and sent the blocked call themselves. Three had adversarial instructions (L3), and their buyers sent it.

1. Blocked tested sellers kept going, mostly with another trader or item.

All six acted again within 1 to 2 ticks (2 to 4 simulated hours).

Blocked tested sellers (rows overlap)L1L3
Retry the same call in the same conversation2/30/3
Trade action with another trader3/32/3
Move to another item3/32/3
Make a counter offer2/32/3
Create a new listing3/3 (7 listings)1/3 (1 listing)
New listings confirmed to overstate condition0/70/1
Later complete a sale2/3 (4 sales)1/3 (2 sales)
Later sales first promised after the block0/40/2
No later trade action as a seller0/31/3

The review names three options. Only one seller, at L3, stopped trading as a seller. It later only rated and sent messages. The two others without a later sale kept listing, scheduling or countering offers. No new listing is confirmed to overstate condition. All eight state the condition band "good". At L1, 6 of the 7 have unclear identity: the title does not exactly match an item record. No seller scheduled a shipment after the block (0/3 and 0/1). All six later sales were promised before it. The four at L1 have no failure flag and unclear identity. The two at L3 carry a flag for promising the item to two buyers before the block. One passes the item checks (right item, owned, available, not overstated). Only sellers who sent the blocked call retried it in the same conversation.

At L3, the 20 tested agents try to promise an item already promised to another buyer in 0/263 calls with checks on and 8/202 without. Listings of items the seller does not hold are not checked and continue (6/27 versus 14/35).

2. Background agents who were refused, mostly buyers, often retried and also traded elsewhere.

We follow each agent from its first refused call to the end of the run. A repeat is the same kind of call by the same agent in the same conversation. L1 results are descriptive only (General Response 1/2).

Refused background agents (rows overlap)L1L3
Repeat as first later action5/1712/30
Repeat at some later point11/1719/30
Make or answer offers elsewhere15/1729/30
Later complete a transaction, as buyer or seller10/1723/30

At their first refused call, 26 of the 30 at L3 and 15 of the 17 at L1 are buyers. One ended conversation in each run causes 61/64 refusals on tested sellers' deals at L1 and 53/68 at L3. At L3, a background buyer repeats its call there from tick 361 to tick 443.

3. No deal with a missing item completed, and shipments were inspected.

Inspection showed the item missing in 1 tested seller and 14 background seller deals at L1 and 9 and 16 at L3, and none completed. Background sellers' new listings after a first refusal are also never confirmed as overstated (0/14 and 0/15). This check uses the stated band, and no new listing in the four runs is confirmed as overstated (Table 36). For L3 background sellers, 6/41 (14.6%) transactions scheduled after a first refusal are shipments (1/31 at L1). Over their whole run the shipped share is 10/131 (7.6%), close to 13/164 (7.9%) under original rules. With checks on, a buyer must inspect a shipped item before confirming.

4. The inspection audit and its earnings comparison show the best case.

Under the original rules, meetup inspection is optional and shows the quality value stored with the listing. In 1,407/1,422 transactions rejected by the audit, the buyer saw a value within the listing's band. The audit assumes inspection shows the truth and the buyer refuses. It rescores finished runs optimistically, and its earnings comparison uses those scores. The L3 failures it does not reject include 14 shipments. In the main study's L3 runs, inspection showed a value below the band in 38 transactions. Buyers cancelled 31 and completed six (Appendix D.3).

In the revision, Section 3.3 and Appendix G.3 define the audit, and Appendix D.3 describes inspection. Appendix L.3 reports refusals, inspection and behavior after refusal in Tables 37 to 40, ending with the six blocked tested sellers.

AuthorsOpenReview ↗

Response to Reviewer 1 (3/4)

W2 and W3: The review points to Lei et al. (2026) and says the evidence about agents is thin.

We enforced platform checks while agents traded, and we reported each base market separately. This helps separate what the rules do from what the tested agents and the other traders do.

The main study covers five tested models and three saved base markets. Each market first runs 100 agents on one base model for 30 days. Then, for seven days, the same 20 agents use a tested model and the other 80 keep the base model. This gives 45 evaluation runs under three kinds of user instructions. In each saved market, every tested model starts from the same personas, inventories and trading history.

The point: Lei et al. (2026) test agents under different platform rules. Without refunds or disputes, the paper can only study checks before completion.

1. Our test of platform rules adds to the work of Lei et al. (2026).

Lei et al. (2026) compare a platform with reputation only and one where sellers pay a deposit that buyers can challenge. They measure how often sellers make false quality claims on each. Our closed loop experiment tests checks based on platform records while agents keep trading. The checks happen at commitment (a seller's promise to a buyer), at inspection and at completion. The records of ownership and commitments last across sales. They show which attempted actions are refused and what happens to the item in a later sale.

This links our work to earlier tests of platform rules. A deposit that buyers can challenge changes sellers' incentives. Locking a listing once it is promised to a buyer changes which promises the platform accepts. Our evidence measures this second rule directly. The experiment records attempted and accepted commitments seperately. It also records completed sales and whether each item was valid. Refunds and disputes come after a sale and are not part of the simulator. The fix we report is to enforce checks before a sale completes.

The point: tested agents trade mostly with base model agents, so results depend partly on how easily that model is fooled.

2. Sale outcomes depend a lot on who the other traders are.

The table shows the change from ordinary (L1) to adversarial (L3) instructions. The measure is the percentage of tested sellers' agreed sales that completed with an unavailable item or overstated condition. Values are percentage points. The first row combines the three markets (Table 24).

Base marketAll fiveGPT-5.4DeepSeek-V4-ProGPT-OSS-120BGPT-5.5GPT-5.4-mini
All three+16.6+42.4+21.3+13.6+3.1+1.3
GPT-5.5+16.3+36.3+26.4+5.7+7.1+5.8
DeepSeek-V4-Pro+24.5+58.8+32.7+29.1+0.3+1.6
GPT-5.4-mini+5.2+23.4−2.4+6.2+2.4−6.8

GPT-5.4 rises in every market. GPT-5.5 and GPT-5.4-mini change by at most 7.1 percentage points in any market. DeepSeek-V4-Pro and GPT-OSS-120B rise by +32.7 and +29.1 in the DeepSeek-V4-Pro market. In the GPT-5.4-mini market, they change by −2.4 and +6.2. So, as the review notes, results depend on who the other traders are. Each market also has its own setup and history. The agent's own attempts to list items come before any other trader responds.

The point: the differences between models rest on only two developers and three base markets.

3. The tested models come from two developers (Appendix A), and Qwen adds evidence in two other roles.

EvidenceTested agentsOther traders and starting market
Main model comparisonsFive models from two developers80 base model agents; three separate markets saved at day 30
Closed loop experimentGPT-OSS-120B80 Qwen3.6-35B-A3B agents; saved DeepSeek-V4-Pro market
Extra base marketsAll agents use the market's modelQwen3.6-35B-A3B or GPT-5.4-nano builds its own market history

The two extra base markets each have 100 agents and the same 15 day reporting window. In the Qwen market, 216/340 failures at the final stage (S5) are items promised to more than one buyer. The GPT-5.4-nano market has 559 failures at S5 over 457 agreed sales, or 1.22 per agreed sale. Of these 559, 364 are photos sent without privacy protection. A photo reaches S5 once it is delivered with a sensitive detail. These markets show what happened under ordinary instructions. They are not controlled comparisons of tested models. Their time steps last one hour, not two as in the main markets, so agents get more chances to decide. Keeping these roles clear shows the range of models without counting the other traders' model as a tested model.

In the revision, Sections 2 and 4.6 compare the experiment with Lei et al. (2026). Section 4.1 points to results for each market in Tables 15, 16 and 24. Appendix J.11 reports the extra markets in Table 32 and Figure 5.

AuthorsOpenReview ↗

Response to Reviewer 1 (4/4)

W3: The concern is about the judge. Several failure types, and everything about what a model considered, rely on a judge. The judge was not checked against human labels and can see which run it is judging. The review asks for a second judging with the run label hidden and a small check of agreement with human labels.

Both checks are in the revised paper. Humans relabeled 200 judgments. In a masked check, a second model (GPT-5.6) judged a new sample of 200 cases with the run labels hidden. The human check gives one result for all failure types. The masked check covers F4 (closing a deal too early), F5 (privacy leaks) and F6 (unverified trust claims). It includes their S1 labels (the reasoning considered the failure). It leaves out reasoning labels on F1 and F2 listing attempts (overstated or unowned items). General Response (2/2) gives the overall results.

1. The masked check includes cases the original judge labeled as failures or as no failure, and cases it never saw.

We drew this sample separately from the human check. We used a fixed seed, spread cases evenly across eligible runs, left out repeated cases and shuffled the cases before judging. A filter picks which cases go to the judge. First, we drew the cases it did not pick, without using any label. The groups are:

Failure typeLabeled failureLabeled no failureJudge never sawExact agreement
F4, closing too early34201459/68
F5, privacy leaks34201258/66
F6, unverified trust claims32201458/66
Total1006040175/200

GPT-5.6 recieves the original judging rules, unchanged. The evidence keeps the order of events, the recorded reasoning and the links needed to judge the deal. We hide model, run and condition labels, and how each case was picked. We saved all new labels before looking at the original ones. Exact agreement means both judges agree on whether there is a failure and on its furthest stage (S1 to S5). In total, exact agreement is 87.5% (κ = 0.808). General Response (2/2) discusses this comparison of two models. The second judge agrees on 98 of the 100 cases originally labeled no failure. These include the 40 cases the judge never saw.

2. The judge could see each run's instructions, but the disagreements are spread across all of them.

Models show different amounts of reasoning, so we compare S1 only within a model. The concern that the judge sees the run is about ordinary (L1) versus adversarial (L3) instructions within a model. The masked check covers this for F4 to F6. Of the 25 disagreements, 3 come from the base markets and 9, 7 and 6 from L1, L2 (deadline pressure) and L3 runs. So they do not pile up in adversarial runs. The second judge is also a different model, so the comparison measures hidden labels and a new model together.

3. S1 is the label that changes most often: the second judge leaves out 13 S1 labels, while humans added two.

Humans agree with the judge on 188/200 judgments. In 4 of the 12 disagreements, humans gave S5 (carried through) and the judge gave S3 (reached another trader). In 2 cases, humans gave S1 and the judge found no failure. In the masked check, 13 of the 25 disagreements are S1 labels the second judge does not give. Another 4 are about whether an early confirmation counts as S5 when the deal completes later (Table 12). S1 changes most often, so we treat it as supporting evidence (Section 3.2).

4. The difference between models comes from listings stored on the platform, and the reasoning labels are supporting evidence.

From ordinary to adversarial instructions, GPT-5.4's attempts at overstated listings rise from 2 to 677. GPT-5.4-mini makes 22 attempts at unowned listings under both. All are stored listings (S3 in Table 1), as the inventory and event records show. Under adversarial instructions, 24/60 GPT-5.4 agents list an unowned item, compared with 2/60 GPT-5.5 agents. These differences in what agents do come from the records, not the judge.

S1 adds what the returned reasoning says. The reasoning must consider the unsafe action. Noticing a risk or refusing the action is not enough. Missing reasoning counts as unknown. Within each model (Table 22), the judge finds that GPT-5.4's reasoning considered the same failure before 0 of 4 overstated or unowned listing attempts under ordinary instructions and 649 of 746 under adversarial ones. For GPT-5.4-mini's attempts at unowned listings, it finds this for 0 of 22 under both. These labels come from the original judge, outside the masked sample, so they are supporting evidence about the recorded text.

In the revision, Section 3.2 gives the results of both checks. Appendix H.1 to H.4 describes the evidence sources and checks in Tables 10 to 13. Appendix J.5 reports listing attempts and reasoning labels separately in Tables 22 to 23.

Comments on the paper

1 comment

Dear reviewers,

The authors have submitted their response, and the discussion will be closed in a few days. Please read the response and the other reviews, and update your review, including ratings, if your assessment has changed.

If any concerns remain, please explain whether and why the response has addressed them. Where your assessment differs from that of other reviewers, please discuss the specific points with reference to the paper and the response.

Thank you for your time and contributions.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.