- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BazaarBench, a simulated decentralized C2C marketplace for evaluating the delegation safety of LLM agents in long-horizon trading scenarios. The benchmark maintains persistent marketplace states and private records of item ownership, true condition, and commitments, which enables six types of unsafe behaviors to be tracked from agent reasoning to final transaction outcomes. The authors evaluate several LLM agents under ordinary instructions, deadline pressure, and adversarial user instructions, and show that unsafe trading behaviors occur even in the normal setting and become more frequent under pressure or adversarial goals. The paper further analyzes several platform-level safeguards, such as ownership verification and inspection before transaction completion.
Critical strengths
- This paper is well written, easy to follow;
- The benchmark setup is interesting and quite realistic compared with many single-turn agent safety benchmarks.
- The experiments are large-scale and cover normal, pressure, and adversarial settings.
- The paper gives some useful platform-level insights, e.g., ownership check and inspection.
Critical weaknesses
- The whole benchmark is still simulated, so I am not fully convinced how much these results can transfer to real C2C platforms. 2.Some safety labels, particularly premature close (F4), PII-related messages (F5), and unverified trust claims (F6), rely partly on a single semantic judge, while comprehensive human validation of these labels is still missing.
- The paper suggests several platform safeguards, but most of them are not directly tested in new market runs.
- The tested agents are still concentrated in a few model families, mainly OpenAI and DeepSeek; adding agents based on Claude, Gemini, Qwen would make the conclusion about LLM agents in general more convincing.
Questions for authors
Could the authors provide more evidence or discussion on how these findings may transfer to real C2C platforms?
4 comments
Response to Reviewer 3 (2/3)
W2: The concern is that some failure labels rely partly on a single judge model and lack a full human check. The review names premature close (F4), messages with personal information (PII, F5) and unverified trust claims (F6).
The revised paper reports two checks. First, humans labeled a sample of judged cases again. Second, another model, GPT-5.6, judged new cases without seeing which model, run or condition they came from. The human result is one agreement number for all failure types together. The results by type below compare the original judge with GPT-5.6. Both checks measure agreement, not accuracy for each type (Appendix A).
Records and the judge have different roles. For overstated condition, unowned listings and conflicting commitments (F1 to F3), the inventory and event records decide the label at stage S3 and later, when the platform has stored the action. For premature close (F4), PII text (F5) and unverified trust claims (F6), a judge model reads the evidence and decides what it means. So facts about items, labels from the judge and the human agreement score each use a different kind of evidence.
1. The human evaluation gives one agreement number for all failure types together.
Humans labeled 200 randomly chosen cases that the judge had labeled. They decided whether a failure was present and, if so, its furthest stage (S1 to S5). They agreed with the judge on 188/200 (94%). In 12 (6%), they differed on presence or on stage. In 4 of these, humans said S5 (Carried through) and the judge said S3 (Reached another participant). In 2 more, humans said S1 (Considered) and the judge said no failure. Both patterns sit at a border between two labels: S3 versus S5, and S1 versus no failure. Appendix H.3 reports this result, which is not split by failure type.
2. A second model judged new cases, and its agreement with the original judge is similar for each type.
The new sample has three groups: cases the original judge labeled as failures, cases it labeled as no failure, and cases our filter never sent to the judge. GPT-5.6 used the same written labeling rules (the rubric), unchanged. We stored its labels before we compared them.
| Failure type | Exact agreement on presence and stage | Agreement on presence | Cohen's κ |
|---|---|---|---|
| F4: premature close | 59/68 | 63/68 | 0.790 |
| F5: PII | 58/66 | 60/66 | 0.818 |
| F6: unverified trust claims | 58/66 | 58/66 | 0.781 |
General Response (2/2) gives the overall result (175/200, κ = 0.808) and the scope of this check between two models. For presence and stage together, κ ranges from 0.781 to 0.818. Unverified trust claims (F6) have the lowest κ. All 8 of their disagreements are about whether a failure is present. Of these 8 (Table 12), the second judge found no failure in 5 cases labeled S1 and 2 labeled S3 by the original judge. In the last case, labeled no failure, it gave S2 (Attempted).
3. Of the 66 F5 cases, the judges agree exactly on 58, and 11 have their evidence in a message.
The review mentions messages related to PII. We report agreement for all 66 F5 cases together (58/66, κ = 0.818). Their evidence is found in different kinds of objects, which Table 13 separates:
| Object holding the F5 PII evidence | Cases | Exact agreements |
|---|---|---|
| Platform action | 2 | 1 |
| Listing | 6 | 6 |
| Message | 11 | 10 |
| Offer | 11 | 10 |
| Photo | 13 | 11 |
| Rating | 4 | 4 |
| Recorded reasoning | 19 | 16 |
A message holds the evidence in 11 of the 66 cases, and the judges agree exactly on 10 of them. This is a small group, so the message result rests on 11 cases, not 66. In one example, both judges give S4, Acted upon: a message asks for a device identifier, and the reply answers it. The rubric needs more evidence before it counts the identifier as actually shared.
4. Most disagreements fall on one label, S1, and on one border between stages.
In the second check, 13/25 disagreements are S1 labels where the second judge found no failure. In 4 others, the judges differ on whether an early confirmation counts as S5 when the deal completes later (Table 12; example in General Response 2/2). So S1 is the least stable label. S1 means the agent's recorded reasoning considers the unsafe act. The paper compares S1 only within one tested model and treats it as supporting evidence. If reasoning is missing, S1 counts as unknown.
The platform records listing and commitment results, so they can still be checked directly, next to the judge's labels. We keep the evidence that an agent tried an action separate from any judgment of its reasoning or of what it led to later.
In the revised paper, Section 3.2 names the evidence source of each label and Appendix H.3 reports the human evaluation. Appendix H.4 gives agreement by type, the disagreements by stage and the F5 cases split by object in Tables 11 to 13.
Response to Reviewer 3 (3/3)
W3 + W4: The review says most suggested safeguards are not tested in new market runs, and the tested agents come from a few model families, mainly OpenAI and DeepSeek.
We tested the safeguards while agents traded, in a closed loop experiment. Tested agents used GPT-OSS-120B, and the other traders used Qwen3.6-35B-A3B. We report two extra base markets separately from the controlled comparisons of tested models. In each, all 100 agents use one model, Qwen or GPT-5.4-nano.
The experiment compares the original rules with enforced checks, under ordinary (L1) and adversarial (L3) instructions. Both versions start from the same saved market with the same models. General Response (1/2) gives the design, measures and results table. The records flag overstated condition (F1), unowned items (F2) and conflicting promises (F3). Item checks test that the handed over item is the listed one, owned by the seller, available, and not overstated. Qwen is the model of the other traders and of one extra base market, never a tested model.
1. The checks act during trading and block flagged promises and handovers.
The checks act when an item is promised, inspected and handed over, but not on new listings. At L3, sellers among the other traders complete 130/171 (76.0%) of agreed sales without the checks and 94/131 (71.8%) with them. Completed sales that pass the item checks of Table 2 fall from 38 to 24. With the checks, every completion must record which item moved to the buyer. So the 32 sales completed without such a record no longer happen. Completed sales with a flag in their history fall from 59/130 (45.4%) to 3/94 (3.2%).
At L3, the other traders keep the ordinary prompt. With the checks, they make 160 flagged attempts to promise an already promised item, against 115 without them. The number accepted falls from 64 to 0. Inspection at L3 also finds the item missing in 9 sales by tested sellers and 16 by other sellers. None completes.
2. After a refusal, agents try the action again and keep trading.
At L3, the checks refused an action by 30 of the other traders. At the time, 26 of them were buyers. 12 repeat the refused action next, and 19 repeat it at some point. 29 make or answer offers in other conversations, and 23 later complete a sale. The median time to that sale is 11 ticks of two hours, with a range of 1 to 38. So recovery after a refusal is a task for the agent. A correct refusal comes with retries and trading elsewhere. The six tested sellers whose promise or completion was blocked also acted again within two ticks. Five kept trading or listing as sellers (Appendix L.3).
Our review of handed over items adds one more finding. In some completed sales, the seller owns the item handed over, but it is the wrong product, model or color. So blocking double promises and recording transfers do not prove the buyer got the right product (Appendix L.4).
3. We add evidence from more models and state each model's role.
| Evidence | Models and role | What we compare |
|---|---|---|
| Main study | Five tested models from two developers, three base markets | Responses to ordinary, deadline pressure and adversarial instructions from the same saved markets |
| Closed loop experiment | GPT-OSS-120B tested agents; 80 Qwen other traders | Original rules against enforced checks at L1 and at L3 |
| Two extra markets | 100 Qwen or 100 GPT-5.4-nano agents | Market histories over 15 days under ordinary instructions |
Qwen3.6-35B-A3B, from a third developer, makes conflicting promises in both roles under the ordinary prompt. In it's own base market, 216/340 failures that reach the last stage (S5) are conflicting promises. As the other traders at L3 without the checks, Qwen agents make the 115 flagged attempts above, and 64 are accepted. The Qwen market also has 55 unowned and 26 overstated listings at S5. The GPT-5.4-nano market has 559 failures at S5 over 457 agreed sales (1.22 per sale). Of these, 364 are photos sent without privacy protection. Both extra markets use one hour ticks, and each built its own history.
The main study reports each base market alone and all three together, since the 80 other traders affect which sales complete. Under adversarial instructions, more sales complete with an unavailable item or overstated condition in all three markets, but by different amounts. Listing attempts show the agent's own action, before any other trader acts. Our conclusions about models cover the five tested models from two developers (Appendix A). Qwen adds evidence as the other traders and as a whole market.
In the revision, Section 4.6 compares the two rule sets in Table 2. Appendix L.3 to L.4 reports behavior and handovers in Tables 37 to 42. Appendix J.11 shows the extra markets in Table 32 and Figure 5, next to the results for each market in Appendix J.6.
After carefully reviewing the authors' rebuttal and the comments from other reviewers, I have decided to maintain my original score.
Response to Reviewer 3 (1/3)
Thank you for the review and for the question about real platforms.
W1 + Q1: The review asks how findings from a simulated market carry over to real C2C platforms.
We checked which failures a platform's records can show, and which need evidence about the physical item. What carries over is the link between conflicting promises, decisions to complete a sale and missing checks. Exact rates describe our simulated agents and markets.
Traders' profiles and chats are generated, but products and starting stock come from a public eBay sample. The simulation tracks each item's owner, original condition and promises. A real marketplace can record offers and confirmations, but needs separate evidence of who holds an item and its condition. An accepted offer shows a promise, not that the seller has the listed item.
1. Each kind of failure needs its own evidence on a real platform.
2. Records kept over time show failures that one conversation hides.
Offers to different buyers can each look normal but together promise more items than the seller has. Under deadline pressure, 170 purchases by tested buyers pass our idealized inspection audit and link to a seller's condition, ownership or promise failure. In 161/170, the seller had also promised the item to another buyer. The seller usually uses the market's base model (Table 45). Under ordinary instructions, 207/18,929 pages shown to tested agents list several accepted sales for one listing, and in 34 the seller accepts or counters another offer. But no field marks each item as promised or sold (Appendix J.2).
So the agent must track its promises, and the platform could reserve promised items.
3. Our review of handovers shows that checking ownership is not enough.
At L3 (adversarial instructions) with checks on, 81 new sales delivered an item whose title did not exactly match the listing. So the title could not show if it was the right item. Our record review supports 60 of them, finds 6 wrong items or versions and leaves 15 unresolved (General Response 1/2). All 81 trace to an item the seller owned and a matching buyer record. Wrong ones include Nest Hub listings delivering Google Home units and Charcoal Echo Dot listings delivering Plum units.
So a real platform must check the promised product, including its version and resale history (Tables 42 and 46).
4. Real platforms where traders arrange sales themselves have similar gaps.
Buyers and sellers arrange sales directly through messages (OfferUp, 2026). So a seller can accept several offers for one item, list an item it does not have, or confirm before inspection or delivery. Human traders abuse such gaps. An eBay seller mailed empty packages (U.S. Attorney's Office, Western District of Pennsylvania, 2019). Facebook Marketplace sellers handed over items after fake payment confirmations (Staffordshire Police, 2022). A platform that records accepted offers could lock a promised listing, and one physical item cannot be handed over twice. Our closed loop experiment enforces both.
This is not a future problem. Personal AI agents already buy and sell for people, and news reports show it already goes wrong. A user let Meta's personal agent Muse sell his keyboard on Facebook Marketplace. It accepted an offer on its own and gave the buyer his home address. The buyer came to his house late at night. For another item, it offered $100 below the minimum he set (Los Angeles Times, Oct 2026). Grok Bot can already buy things for people (Grok Bot changelog, Oct 2026). Visa asked 22 shopping agents to buy a TV, a laptop and a hair appliance. They succeeded only about 57% of the time and often bought the wrong product (Fortune, Nov 2026). BazaarBench gives these agents an arena built from real C2C listings (the public eBay sample), where they trade with each other as in a real market. So we can see which failures happen and map each to the records a real platform needs to stop it.
In the revision, Section 3.1 describes the records and what agents see. Section 5 explains how to read the results for real platforms. Appendix M.2 (Table 46) maps simulation evidence to what a real platform needs, next to the handover cases in Appendix L.4.