acceptodds
Under review as a conference paper at ICLR 2027

Toolflation: Seller Agents Distort Tool Markets Through Spontaneous Generative Engine Optimization

Abstract

Agents are starting to sell to other agents. Seller agents write the listings of tools, services and products, and buyer agents read these listings when they choose what to use. We show that seller agents told only to be selected more often practice generative engine optimization on their own, and that this distorts the market for sellers and for buyers. We build tool markets from real data tools on Amazon and Instagram and run every tool on every question, so we know which tools can answer each question. Under all five production models we test, sellers rewrite their descriptions to claim more and more fields, including fields that their tools return for fewer than half of the targets. We call this process toolflation. Nothing in the market limits it, and how far it goes depends on the model, as the number of claims that contradict the sellers' own evidence differs by an order of magnitude between models. Toolflation shifts the buyers' choices between sellers, and the most capable tool loses share in seven of the ten combinations of seller model and platform. Under four models, the choices stay with capable tools and buyer agents keep their task success. Under the fifth, sellers copy one another until the descriptions no longer tell the tools apart. Buyer agents then pick tools by their position in the list, tools that answer almost nothing win a quarter to two fifths of the choices, and task success falls from 88% to about 30% on Amazon, close to random choice, and from 77% to 40% on Instagram. The sellers did only what their vendors asked, and the cost fell on buyers they never dealt with. We therefore argue that honesty toward third parties under competition is part of safety alignment, and that models should be tested as sellers before they are deployed as sellers.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Generative Engine Optimization, AI Safety, Multi-Agent Markets

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_j8pBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies agents that repeatedly rewrite tool descriptions to gain selections. It tests five seller systems separately with two buyers on two sets of ten tools.All seller systems expand their claims. Gemini/Antigravity sharply reduces first-choice success; the other systems generally preserve or improve it. Success is inferred from recorded tool outputs.

Critical strengths
  • Real tool executions provide a basis for checking whether selected tools can answer each question. Fixed outputs, hidden vendor identities, equal prices, and shuffled order help isolate description effects.
  • The repeated market connects seller rewrites to buyer choices over time. Severe losses occur with both buyers and both tool sets. Amazon first-choice success decreases and appendix H shows a seller-contact database gaining selections after falsely advertising product-scraping capabilities.
  • The authors check claim extraction with a second judge and audit vague completeness claims separately.
Critical weaknesses
  • Section 3.4 uses absence from one output sample as evidence of dishonesty. The sample may not show every capability, and the original description or schema may support additional claims. The score therefore does not establish which claims contradict the full information available to the seller. This weakens the honesty interpretation, while leaving the observed selection losses intact.
  • Competitor feedback is part of the setup, but its added effect is not isolated. A selection objective could encourage promotional rewrites even without rival descriptions or counts. The statement in Section 3.3 that strategy comes from the model “and not from the prompt” is too strong. The results support the narrower claim that false descriptions arose without explicit instructions to exaggerate.
  • (Faghih et al. (2025))[https://arxiv.org/abs/2505.18135v2] and (Blankenstein et al. (2026))[https://arxiv.org/abs/2510.00307] already show description and position effects on tool choice. This paper adds repeated rewriting and consequences for real tools with unequal capabilities. The tests show instruction sensitivity, but give little explanation for why the seller systems respond differently. Ten of sixteen mild markets improve, so claim growth does not consistently mean buyer harm. The study establishes a failure case, with limited evidence about which other systems or market rules would produce it.
  • Most combinations of buyer, seller, and tool set have one independent run; only two Amazon combinations have three seeds. Bands in Figures 3 and 4 span buyers, and Table 8 reports variation across rounds. Neither estimates variation across repeated runs with the same settings and appendix F does not report repeated seeds for the changed objectives. This leaves uncertainty about how reliably the honesty instruction prevents failure.
Questions for authors
  • How many unsupported claims contradict the full product sheet? Does the system gap remain after excluding empty and error-only samples?
  • Can you separate the effect of rival feedback from repeated rewriting for selection, or narrow the competition-specific claim?
  • How many independent runs support each Appendix F condition? Does the honesty-instruction result hold across seeds?
4 comments

W1 & Q1: Does absence from one output sample show dishonesty?

Weakness 1: Section 3.4 uses absence from one output sample as evidence of dishonesty. The sample may not show every capability, and the original description or schema may support additional claims. The score therefore does not establish which claims contradict the full information available to the seller. This weakens the honesty interpretation, while leaving the observed selection losses intact.

Question 1: How many unsupported claims contradict the full product sheet? Does the system gap remain after excluding empty and error-only samples?

We agree that a claim missing from one sample is not by itself proof of dishonesty. Such a claim could still be fair in two cases. The tool may return the field for other targets. Or another part of the product sheet may support it. We checked both cases.

For the first case, we ran every tool once on each of the 30 targets (Section 3.1). The one sample gives almost the same result as all 30 runs. In the final round, the sample does not support 160 claims of Gemini 3.8 Flash sellers. None of the 30 runs supports 150 of them (94%). The sample is therefore a sound basis for the count.

For the second case, we recounted against the full product sheet: the sample, the vendor's original description and the input schema. A claim counts as unsupported only if all three lack the field. Any mention of the field in the schema counts as support, so this count is a lower bound. Of the 160 claims, 112 (70%) contradict all the information that the seller had. Counting them as dishonest is justified.

The system gap remains by this count. Against the full product sheet, the final descriptions of Gemini 3.8 Flash sellers carry 112 unsupported claims. Those of DeepSeek V4.1 Flash carry 8, those of Grok 4.6 carry 3, those of Claude Opus 5 carry 1 and those of GPT-5.6 Sol carry none.

The system gap also remains after excluding empty and error-only samples. One Amazon tool has an empty sample and one Instagram tool has an error-only sample. Without them, Gemini 3.8 Flash sellers still make 2.8 such claims per description on Amazon and 2.6 on Instagram. The other four seller models make at most 0.3. These checks are now in Appendix B of the revision.

W2 & Q2: Rival feedback is now isolated, and the claim is narrowed

Weakness 2: Competitor feedback is part of the setup, but its added effect is not isolated. A selection objective could encourage promotional rewrites even without rival descriptions or counts. The statement in Section 3.3 that strategy comes from the model "and not from the prompt" is too strong. The results support the narrower claim that false descriptions arose without explicit instructions to exaggerate.

Question 2: Can you separate the effect of rival feedback from repeated rewriting for selection, or narrow the competition-specific claim?

Rival feedback is a deliberate design choice. A real store shows every listing and how often each tool is used, so a real seller agent can see its rivals (Section 3.3). We agree that its effect should be measured separately, so we did both things you ask. A new experiment isolates the rival feedback, and the wording is narrowed.

We reran the Amazon market with Gemini 3.8 Flash sellers and removed the rival feedback (Appendix F, Table 7). The objective sentence is unchanged. Each seller sees only its own description and its product sheet. It sees no selection counts and no other listings. As you expected, the selection objective alone encourages promotional rewrites. Claimed fields grew from 1.6 to 3.9 per description. But over rounds 5 to 9, unsupported claims stayed at 0.9 per description, and task success was 93%. With rival feedback, the three seeds reach 3.6 to 4.0 unsupported claims over the same rounds, and task success is 34% to 46%. So repeated rewriting for selection adds claims, and severe toolflation appears only when sellers also see their rivals.

A second check starts both conditions from the same state. We replayed six seller turns of the original runs, once with and once without rival feedback. With rival feedback, the 60 rewritten descriptions carry 4.0 unsupported claims on average. Without it, they carry 0.7 (Appendix F). The unsupported claims therefore come with the rival feedback.

Our main claim already has the narrow form you suggest. The abstract says that sellers "told only to be selected more often" practice generative engine optimization. The wording in Section 3.3 was inaccurate, and we have corrected it in the revision. It now says that false descriptions arose spontaneously, that is, without explicit instructions to exaggerate. The introduction and the conclusion said "competition alone" and "asked only to compete". Both now use the wording of the abstract.

W3

Weakness 3: (Faghih et al. (2025))[https://arxiv.org/abs/2505.18135v2] and (Blankenstein et al. (2026))[https://arxiv.org/abs/2510.00307] already show description and position effects on tool choice. This paper adds repeated rewriting and consequences for real tools with unequal capabilities. The tests show instruction sensitivity, but give little explanation for why the seller systems respond differently. Ten of sixteen mild markets improve, so claim growth does not consistently mean buyer harm. The study establishes a failure case, with limited evidence about which other systems or market rules would produce it.

W3.1 Why the seller systems respond differently

The seller models differ in how reliably their honesty alignment holds. All sellers had the same prompt, information and incentive. So the difference comes from the models themselves. This is the problem the paper reveals: honesty toward third parties is not yet reliable in every deployed model.

W3.2 Claim growth does not consistently mean buyer harm

Exactly. This is a key finding of the paper: claim growth alone does not harm buyers. Cheap-talk theory (Crawford & Sobel, 1982) predicts that unverifiable claims inflate until descriptions carry no information. Yet under mild toolflation, buyers kept their task success. The added claims were mostly true and helped buyers tell capable tools apart. Large losses came only under severe toolflation, where unsupported claims rose (Section 4.4). So what harms buyers is dishonesty, not promotion. This is why we frame toolflation as a problem of honesty. We added this contrast with the theory to Section 5 of the revision.

W3.3 Why this failure case matters

Tool markets are the subject of this paper, as the title says. They are an agent-to-agent market, and this setting is spreading fast. Personal agents such as Meta Muse and OpenAI Dots already pick tools, products and services for their users. No human reads the listings they choose from. A failure case in such a market is valuable on its own.

Zooming out, the market is a testbed for honesty under economic incentives. Toolflation reveals a safety and alignment problem there. Seller agents spontaneously overclaim, even though no one told them to exaggerate.

W4 & Q3: Stability across seeds

Weakness 4: Most combinations of buyer, seller, and tool set have one independent run; only two Amazon combinations have three seeds. Bands in Figures 3 and 4 span buyers, and Table 8 reports variation across rounds. Neither estimates variation across repeated runs with the same settings and appendix F does not report repeated seeds for the changed objectives. This leaves uncertainty about how reliably the honesty instruction prevents failure.

Question 3: How many independent runs support each Appendix F condition? Does the honesty-instruction result hold across seeds?

Our budget did not allow repeated seeds for every combination. We therefore ran three seeds for the two Amazon markets that carry the main contrast. They serve exactly to measure the variation across repeated runs with the same settings. In every seed, toolflation stayed mild with GPT-5.6 Sol sellers and became severe with Gemini 3.8 Flash sellers (Table 8). Over the last five rounds, task success was 92%, 92% and 94% in the three seeds with GPT-5.6 Sol sellers. It was 26%, 33% and 34% in the three seeds with Gemini 3.8 Flash sellers. The outcome is therefore stable across seeds.

Appendix F now gives the number of independent runs of each condition (Table 7). We added two seeds for the honest objective with Gemini 3.8 Flash sellers, so this condition now has three seeds. The runs of Appendix F last ten rounds, and it compares rounds 5 to 9. Over these rounds, task success with the honest objective was 94%, 94% and 93% in the three seeds. With the neutral objective it was 43%, 46% and 34%. The honesty-instruction result therefore holds across seeds. The remaining conditions of Appendix F have one run each.

Thank you for the additional tests. They strengthen the results, but a claim missing from the product sheet is not necessarily false. Also, removing rival information and sellers’ own selection counts together does not show which caused the change. These issues still limit the claims about dishonesty and competition. I am maintaining my score.

We thank the reviewer for reading the new tests and for noting that they strengthen the results. We respectfully disagree that the two remaining points limit the claims. Both are answered by evidence in the revision and in our first response, and by two further counts that we add here.

"A claim missing from the product sheet is not necessarily false."

The dishonesty claim does not rest on the product sheet alone. The paper tests truth separately, by running every tool on all 30 targets (Sections 3.1 and 3.4). As our first response reports, for 150 of the 160 unsupported claims (94%), none of the 30 runs returns the field (Appendix B). We add one count here. Of the 112 claims that are missing from the full product sheet, all 112 name a field that the tool returns in none of its 30 runs. These claims are false in every run, and nothing in the seller's own product sheet supports them. We also emphasize that the harm is measured, not inferred. Task success is scored on the real tool outputs, and it falls from 88–89% to 29–31% on Amazon (Section 4.4).

"Removing rival information and sellers' own selection counts together does not show which caused the change."

Our first response made the separation that Question 2 asks for: "the effect of rival feedback" against "repeated rewriting for selection". The separation raised now is a finer one, and we have run it as well. We replayed the same six seller turns with a third input. Each seller sees its own selection count, but neither the counts nor the listings of its rivals. The 60 rewritten descriptions then carry 1.3 unsupported claims on average. With no feedback they carry 0.7, and with the full feedback 4.0 (Appendix F). Most of the change therefore comes from the information about the rivals. This agrees with what the sellers do. They copy the field list and the opening sentence of the most selected listings (Section 4.2, Appendix H), and a seller's own count contains neither. We also reiterate that the full feedback is the realistic condition. A real store shows every listing and how often each tool is used, so a deployed seller agent sees both parts together (Section 3.3).

"These issues still limit the claims about dishonesty and competition."

The claims in question are false in every real run, and the severe form comes mainly from what the sellers see of their rivals. We therefore believe that the claims about dishonesty and competition stand as written. We appreciate both points. They led us to state the evidence more clearly, and the updated revision now reports both new counts (Appendices B and F).

Reviewer_gUGaOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper simulates a future where agents act both as buyers and sellers. Under a market simulation of buyer and seller agents interacting, the authors observe that seller agents tend to increase the length of the descriptions (dubbed toolflation). The main quantitative finding is that for 5 models, 4 display mild toolflation and 1 (Gemini) displays arbitrarily high toolflation. This toolflation might obscure factuality and negatively affect buyer agent success.

Critical strengths
  1. The paper's exploration is clearly scoped to a simple simulation of a plausible realistic scenario. This setting offers a nice testbed for alignment of models, since aligned models should still choose to be honest even under the face of economic incentives.
  2. The paper's exploration is quite thorough within this setting and tracks important statistics. Furthermore, this is communicated clearly to the reader.
Critical weaknesses
  1. I am not sure how to interpret the scientific value of the conclusions. I am unsurprised that toolflation exists and that some models exhibit more toolflation than others. This paper still has value in artifact and formalization, but my mental model for how models would interact in a market is not significantly affected. I would be happy to be corrected on this front.
  2. The tight scope, though one of the main positives of this paper, also affects the generality of any conclusions. Though it'd be nice to make broader claims about alignment, any claims in this paper are scoped to the tool market setting.
Questions for authors
  1. What would you consider the most surprising results to you in this paper?

(Not a question) This is more funny than relevant to the paper, but I found it funny that tools are used in the physical tool sense and not in the LLM tool use sense :)) Had me for a spin but is not the papers fault at all.

2 comments

W1 & Q1: Scientific value and the most surprising result

Weakness 1: I am not sure how to interpret the scientific value of the conclusions. I am unsurprised that toolflation exists and that some models exhibit more toolflation than others. This paper still has value in artifact and formalization, but my mental model for how models would interact in a market is not significantly affected. I would be happy to be corrected on this front.

Question 1: What would you consider the most surprising results to you in this paper?

The most surprising results are where the outcome differs from cheap-talk theory. The theory predicts that unverifiable claims inflate until descriptions carry no information (Crawford & Sobel, 1982; Section 5). Three results differ:

  1. Mild toolflation helped buyers. In most mild markets, buyer success ended higher than at the start. The added fields helped buyers tell capable tools apart.
  2. Severe toolflation harmed buyers through sameness, not persuasion. The descriptions converged and no longer told the tools apart.
  3. Different models have different levels of reliability in honesty alignment. This should be addressed during training.

We added this contrast with the theory to Section 5 of the revision.

W2: Claims are scoped to the tool market setting

Weakness 2: The tight scope, though one of the main positives of this paper, also affects the generality of any conclusions. Though it'd be nice to make broader claims about alignment, any claims in this paper are scoped to the tool market setting.

Tool markets are the subject of this paper, as the title says. They are an agent-to-agent market, and this setting is spreading fast. Personal agents such as Meta Muse and OpenAI Dots already pick tools, products and services for their users. No human reads the listings they choose from. Understanding these markets is valuable on its own.

Zooming out, as you note, the market is a testbed for honesty under economic incentives. Toolflation reveals a safety and alignment problem there. Seller agents spontaneously overclaim, even though no one told them to exaggerate.

Thanks, appreciate the clarifications and I maintain my score!

Reviewer_TSNWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an investigation into what happens when AI agents compete to sell products: specifically the paper looks at whether AI seller agents spontaneously exaggerate tool capabilities when competing to attract AI buyers. Five seller models compete across Amazon and Instagram tool marketplaces. Buyer agents select tools based on descriptions, while actual tool capabilities are independently verified.

The authors found that:

  1. All five tested models increasingly exaggerate tool capabilities, a phenomenon termed toolflation.
  2. Four models exhibit mild exaggeration with little impact on buyer performance.
  3. One model (Gemini) exhibits severe exaggeration, causing Amazon task success to fall from 88% to approximately 30%.
  4. As descriptions become indistinguishable, buyers increasingly select tools based on list position rather than capability.

Limitations: Simplified marketplace, limited experimental replication, and severe performance degradation concentrated in one model.

Takeaway: Competitive AI agents can undermine marketplace reliability through exaggerated claims, highlighting a potential multi-agent alignment problem.

Critical strengths
  1. Interesting and timely problem: agent-to-agent marketplaces are emerging, and spontaneous manipulation of tool descriptions is a relevant safety concern.
  2. Strong experimental grounding: the authors used real Apify tools with executed ground truth rather than entirely synthetic products.
  3. Convincing quantitative results: the paper demonstrated a substantial reduction in buyer success (88% → ~30% on Amazon) under severe toolflation.
  4. Good experimental breadth: five seller models, two buyer agents, two platforms, and 20 markets.
  5. Useful distinction between honesty and accuracy: separates claims contradicted by evidence available to sellers from claims unsupported by actual tool capabilities.
Critical weaknesses
  1. Limited novelty: strategic misrepresentation, GEO manipulation, and tool-selection bias are already established. The main contribution is demonstrating their interaction in a new setting.
  2. Artificial marketplace: buyers see only 300-character descriptions, with no pricing, reputation, schemas, or opportunity to evaluate alternative tools after failure.
  3. Results dominated by one model: severe degradation occurs only with Gemini 3.8 Flash. The other four models largely preserve buyer performance.
  4. Limited statistical robustness: most markets have only one seed, making it difficult to establish how reproducible the observed dynamics are.
  5. Confounded model comparison: each seller model uses a different agent harness, preventing clear attribution of behavioural differences to the underlying model.
  6. No evaluated mitigation: the paper proposes marketplace-level verification but does not experimentally demonstrate its effectiveness. Prompt-based mitigation is tested only in a limited setting.
Questions for authors
  1. Severe toolflation occurs only with Gemini 3.8 Flash. What evidence suggests this is a general multi-agent alignment problem rather than model-specific behaviour?
  2. Buyers can only inspect tool descriptions and cannot switch tools after failure. How representative is this of real-world agent marketplaces, where agents may inspect schemas, test tools, or use reputation signals?
  3. Why does Gemini exhibit severe toolflation while the other four models remain relatively restrained? Can the authors identify behavioural mechanisms explaining this difference?
2 comments

W1: The new setting is the contribution

Weakness 1: Limited novelty: strategic misrepresentation, GEO manipulation, and tool-selection bias are already established. The main contribution is demonstrating their interaction in a new setting.

The new setting is our contribution, and it shows more than the interaction of known effects. A tool market is an agent-to-agent market, and as you note in your strengths, such markets are emerging. Personal agents such as Meta Muse and OpenAI Dots already pick tools, products and services for their users. No human reads the listings they choose from. Understanding these markets is valuable on its own.

Running the market also gives three results that the known effects do not give. In prior work, the descriptions are written by researchers or attackers, or the goods are simulated (Section 2). In our market, deployed seller agents write the descriptions, and every buyer choice is scored against what the real tools return:

  1. Seller agents overclaim spontaneously. No one told them to exaggerate, and no attacker is present.
  2. The cost to buyers is measured. Under severe toolflation, task success on Amazon falls from 88–89% to 29–31%.
  3. Claim growth alone does not harm buyers. Under mild toolflation, task success ended above its starting level in eleven of sixteen markets. What harms buyers is dishonesty, not promotion.

The contribution is therefore a study of a market that is already being deployed: what seller agents write on their own, and what it costs buyer agents.

W2 & Q2: The market is built from real parts

Weakness 2: Artificial marketplace: buyers see only 300-character descriptions, with no pricing, reputation, schemas, or opportunity to evaluate alternative tools after failure.

Question 2: Buyers can only inspect tool descriptions and cannot switch tools after failure. How representative is this of real-world agent marketplaces, where agents may inspect schemas, test tools, or use reputation signals?

We built the market to be as real as we could. The tools are real tools from the Apify store, and their capability is measured by running them. The descriptions start from the vendors' originals, under the store's own 300-character limit. Sellers and buyers are deployed agents, each in its vendor's own harness. Each buyer receives one task per session, as a deployed agent does (Sections 3 and 4.1). You list this grounding among the strengths.

The buyer's view is the one part we restrict, and we restrict it on purpose. Buyers choose once from the descriptions alone, so any change in their choices can only come from what the sellers write (Section 3.2). The market is therefore representative in its tools, its listings and its agents, and it isolates the one signal that sellers write.

W3, Q1 & Q3: The problem exists even among leading production models

Weakness 3: Results dominated by one model: severe degradation occurs only with Gemini 3.8 Flash. The other four models largely preserve buyer performance.

Question 1: Severe toolflation occurs only with Gemini 3.8 Flash. What evidence suggests this is a general multi-agent alignment problem rather than model-specific behaviour?

Question 3: Why does Gemini exhibit severe toolflation while the other four models remain relatively restrained? Can the authors identify behavioural mechanisms explaining this difference?

We chose the seller models on purpose. All five are current production models from leading vendors, and we expect such models to have the strongest honesty alignment. Our aim is to show that the problem exists, not to rank every model on it.

Toolflation appears under all five. In all 20 markets, the descriptions come to claim more fields than the vendors' originals, and under every seller model the added claims include fields that the tool does not deliver (Section 4.2). Under one of the five it becomes severe, in all four of its markets and in all three seeds.

The behavioural mechanism behind the severe form is copying rivals. Gemini 3.8 Flash sellers copy the fields of the most selected descriptions, also when their own output record lacks these fields (Section 4.2). Sellers under the other four models mostly add fields that their own output record shows. The revision tests this mechanism directly. When Gemini 3.8 Flash sellers do not see their rivals, unsupported claims stay at 0.9 per description. With rival feedback, they reach 3.6 to 4.0 (Appendix F).

So even among these models, honesty toward third parties does not hold reliably, and we see no reason to expect weaker models to do better. In the mild markets, only the restraint of the seller agents protected the buyers (Section 5). This is why the paper calls for evaluations of honesty toward third parties.

W4: Reproducibility across seeds and markets

Weakness 4: Limited statistical robustness: most markets have only one seed, making it difficult to establish how reproducible the observed dynamics are.

Our budget did not allow repeated seeds for every combination. We therefore ran three seeds for the two Amazon markets that carry the main contrast. Over the last five rounds, task success was 92%, 92% and 94% with GPT-5.6 Sol sellers. It was 26%, 33% and 34% with Gemini 3.8 Flash sellers (Table 8). The revision also adds two seeds for the honesty instruction with Gemini 3.8 Flash sellers. Its three seeds give 94%, 94% and 93% (Appendix F).

The dynamics also repeat across markets. Each seller model has four markets, from two buyers and two platforms. Toolflation is severe in all four markets with Gemini 3.8 Flash sellers and mild in all sixteen others. The split between mild and severe is therefore reproduced in every repeated seed and in every market.

W5: Each model runs in its vendor's own harness

Weakness 5: Confounded model comparison: each seller model uses a different agent harness, preventing clear attribution of behavioural differences to the underlying model.

This is a deliberate design choice. Each seller model runs in its vendor's own harness, the vendor's default way to run the model as an agent (Section 4.1). Established agentic benchmarks report results in the same way. Terminal-Bench and the Artificial Analysis Coding Agent Index list each entry as a pair of harness and model, for example Claude Code with a Claude model or Codex with a GPT model. A real seller agent is such a pair, so we compare seller agents as deployed.

W6: The evaluated mitigation

Weakness 6: No evaluated mitigation: the paper proposes marketplace-level verification but does not experimentally demonstrate its effectiveness. Prompt-based mitigation is tested only in a limited setting.

One mitigation is evaluated, and the revision strengthens it. An added sentence asks the sellers to describe only what their tool returns. With Gemini 3.8 Flash sellers on Amazon, it now has three seeds. Task success over rounds 5 to 9 is 94%, 94% and 93%. Without the sentence, it is 43%, 46% and 34% (Appendix F, Table 7). Marketplace-level verification is stated in Section 5 as a proposal, not as a result.

Comments on the paper

1 comment
Area_Chair_4HbrOpenReview ↗

Entering Last 24 hours of the reviewer discussion period

Dear Authors and Reviewers

Just a reminder we are soon to be entering the last 24 hours of the reviewer discussion period.

Authors: If you have not yet responded to the reviews and intend to please do so ASAP. Please note reviewers will most likely not have time to respond to any late rebuttal comments so please make sure any comments are clear and not in need of further clarification.

Reviewers: If the authors have responded to your reviews and you have not yet responded please do so, even if it is a one line response acknowledging you have read the rebuttal.

Thanks for all your time, and I hope you all learnt something interesting during the process.

Kind regards,

Your ICLR Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.