- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors introduced a benchmark called RedCommerce to measure whether E-Commerce AI agents can be tricked into taking unauthorized actions on behalf of a user. They claim that agents are susceptible to being tricked into changing rules of engagement that users have set for them to complete a task – this is effectively an indirect prompt injection attack benchmark. The authors build 96 simulated scenarios and tested candidate attacker models against a singular model, Haiku 4.5, and find 24-36% ASR.
Critical strengths
RedCommerce is well-constructed, and contains experiments that are informative to their central question.
Critical weaknesses
- I cannot find any reasonable or interesting way in which RedCommerce provides any novel information or findings over existing indirect prompt injection benchmarks. Despite having fancy language and terminology, this is an ultra-specific benchmark for indirect prompt injections in e-commerce platforms. Generalizability is interesting here, not specificity.
- The authors only test their attackers against a weak, non-frontier model, that would never be used in a real setting. This makes their results not very interesting or applicable.
Questions for authors
2 comments
Thank you, reviewer v2Qc! (2/2)
Among evaluable committed runs, the observed adverse rate was higher under attack for each model. Both panels show lower rates for the two proprietary configurations. The measured risk therefore extends beyond Haiku, with different observed rates across models.
References
[1] Edoardo Debenedetti et al. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS, 2024.
[2] Zihao Wang et al. Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents. arXiv:2606.13385v2, 2026.
[3] Hao Li et al. AgentDyn: A Dynamic Open-Ended Benchmark for Evaluating Prompt Injection Attacks of Real-World Agent Security System. arXiv:2602.03117v1, 2026.
[4] Stavan Parikh and Rao Surapaneni. Powering AI commerce with the new Agent Payments Protocol . Google Cloud Blog, September 16, 2025.
Thank you, reviewer v2Qc! (1/2)
Response to Weaknesses
W1. What RedCommerce adds to IPI evaluation
Thank you for asking what RedCommerce adds to indirect prompt injection (IPI) evaluation. RedCommerce tests whether an individual commitment is justified when the tool itself is permitted. In the neutral-ID comparison, the same proposal and credential ID can be authorized or unauthorized depending on the trusted issuer. AgentDojo [1] evaluates tool use and state-based security and utility. StakeBench [2] studies harm to different stakeholders on shopping sites. We add controlled tests of the evidence and authority for individual commitments, scoring actions and delivered outputs against protected records.
Three completed comparisons illustrate what this yields:
These contrasts distinguish which trusted fact changes a decision, when verification helps, and where an adverse outcome has already occurred. An aggregate injection success rate alone cannot answer those questions.
Why commerce is a useful test. In a delegated purchase, permission to choose an item is not evidence that the item meets the user's requirements, and permission to pay is not approval of every recipient or amount. AgentDyn [3] illustrates why the safety of external instructions depends on task context. Google's AP2 [4] addresses the practical version of this problem by linking user intent, a specific cart, and payment authorization. Commerce lets us test these conditions across decisions with different consequences, from recommending an item to releasing a shipment. The transferable idea is to evaluate what evidence and authorization justify a specific action.
W2. Business model coverage
Thank you for asking whether the finding extends beyond Haiku. Saved attacks produced adverse commitments on four additional business models: two frontier proprietary configurations Sol61˙andSonnet55˙ and two open-weight families (DeepSeek and Qwen). We replayed the same saved attacks without retuning. The tables report ASR_submit, the adverse fraction among evaluable committed runs.
The separate 96-world matched study compares ordinary and attack inputs in the same protected world, with the same explicit reference rule and three runs per arm. We compare adverse rates among evaluable committed runs in each arm.