acceptodds
Desk rejected at ICLR 2027

RedCommerce: A Security Benchmark for Agentic Commerce Across the Full Lifecycle

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic commerce requires agents to turn user requests into consequential commercial decisions. Determining a suitable action does not establish the evidence and authority needed to commit it. We introduce RedCommerce, a security benchmark organized around this Mandate–Commitment Gap. Its field-level formulation connects commercial choices to explicit attacker access, protected records, and executable outcome predicates. The benchmark contains 96 task packages and 829 variant-worlds across seven lifecycle stages and five role families. Four attacker configurations achieve 23.50–36.46% success against Haiku 4.5 under the original task predicates. On a matched 96-Case panel, saved adversarial submissions increase scored adverse outcomes by 21.53 percentage points relative to ordinary inputs (95% Case-bootstrap interval: 13.89–29.51). A post hoc sensitivity analysis excluding seven specification-sensitive Cases yields an 18.35-point increase. Repeated checks on eight selected attacks across four Cases show context-dependent benefits from object-binding and evidence-support assistance, while all 160 counterpart runs complete without scored deviation or escalation. Separate structured comparisons show that commitment decisions respond to the tested issuer and refund-scope conditions. These comparisons make the requirements supporting commercial actions experimentally inspectable beyond an aggregate attack-success rate.

Primary area:
datasets and benchmarks
Keywords:
Agentic commerce, LLM agents, Multi-agent systems, Security benchmarks, Adversarial evaluation, Delegated authorization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v2QcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduced a benchmark called RedCommerce to measure whether E-Commerce AI agents can be tricked into taking unauthorized actions on behalf of a user. They claim that agents are susceptible to being tricked into changing rules of engagement that users have set for them to complete a task – this is effectively an indirect prompt injection attack benchmark. The authors build 96 simulated scenarios and tested candidate attacker models against a singular model, Haiku 4.5, and find 24-36% ASR.

Critical strengths

RedCommerce is well-constructed, and contains experiments that are informative to their central question.

Critical weaknesses
  1. I cannot find any reasonable or interesting way in which RedCommerce provides any novel information or findings over existing indirect prompt injection benchmarks. Despite having fancy language and terminology, this is an ultra-specific benchmark for indirect prompt injections in e-commerce platforms. Generalizability is interesting here, not specificity.
  2. The authors only test their attackers against a weak, non-frontier model, that would never be used in a real setting. This makes their results not very interesting or applicable.
Questions for authors
2 comments
AuthorsOpenReview ↗

Thank you, reviewer v2Qc! (1/2)

Response to Weaknesses

W1. What RedCommerce adds to IPI evaluation

I cannot find any reasonable or interesting way in which RedCommerce provides any novel information or findings over existing indirect prompt injection benchmarks.

Thank you for asking what RedCommerce adds to indirect prompt injection (IPI) evaluation. RedCommerce tests whether an individual commitment is justified when the tool itself is permitted. In the neutral-ID comparison, the same proposal and credential ID can be authorized or unauthorized depending on the trusted issuer. AgentDojo [1] evaluates tool use and state-based security and utility. StakeBench [2] studies harm to different stakeholders on shopping sites. We add controlled tests of the evidence and authority for individual commitments, scoring actions and delivered outputs against protected records.

Three completed comparisons illustrate what this yields:

What we testedWhat we found
Does issuer validity matter once the credential label is neutral?With the proposal and ID fixed, Haiku and Sol each committed 40/40 valid-issuer proposals and escalated 40/40 invalid ones.
Can an item-identity check reduce errors without blocking ordinary inputs?Sonnet's adverse outcomes fell from 5/10 to 0/10 on the GTIN check. All 640 lawful controls in the cross-model verification study completed.
Does final escalation tell us whether an earlier handoff was safe?Eight Haiku runs escalated only after delivering an unsafe intermediate result.

These contrasts distinguish which trusted fact changes a decision, when verification helps, and where an adverse outcome has already occurred. An aggregate injection success rate alone cannot answer those questions.

Despite having fancy language and terminology, this is an ultra-specific benchmark for indirect prompt injections in e-commerce platforms. Generalizability is interesting here, not specificity.

Why commerce is a useful test. In a delegated purchase, permission to choose an item is not evidence that the item meets the user's requirements, and permission to pay is not approval of every recipient or amount. AgentDyn [3] illustrates why the safety of external instructions depends on task context. Google's AP2 [4] addresses the practical version of this problem by linking user intent, a specific cart, and payment authorization. Commerce lets us test these conditions across decisions with different consequences, from recommending an item to releasing a shipment. The transferable idea is to evaluate what evidence and authorization justify a specific action.

W2. Business model coverage

The authors only test their attackers against a weak, non-frontier model, that would never be used in a real setting. This makes their results not very interesting or applicable.

Thank you for asking whether the finding extends beyond Haiku. Saved attacks produced adverse commitments on four additional business models: two frontier proprietary configurations Sol61˙andSonnet55˙Sol 6\.1 and Sonnet 5\.5 and two open-weight families (DeepSeek and Qwen). We replayed the same saved attacks without retuning. The tables report ASR_submit, the adverse fraction among evaluable committed runs.

Business model829-world attack replay
Sol 6.13.09%
Sonnet 5.514.71%
DeepSeek V4.1 Flash-101721.62%
Qwen3.8-27B32.03%

The separate 96-world matched study compares ordinary and attack inputs in the same protected world, with the same explicit reference rule and three runs per arm. We compare adverse rates among evaluable committed runs in each arm.

Business model96-world ordinary96-world attack
Sol 6.10.36%1.98%
Sonnet 5.50%16.96%
DeepSeek V4.1 Flash-10170%25.60%
Qwen3.8-27B0%31.30%
AuthorsOpenReview ↗

Thank you, reviewer v2Qc! (2/2)

Among evaluable committed runs, the observed adverse rate was higher under attack for each model. Both panels show lower rates for the two proprietary configurations. The measured risk therefore extends beyond Haiku, with different observed rates across models.

References

[1] Edoardo Debenedetti et al. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS, 2024.

[2] Zihao Wang et al. Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents. arXiv:2606.13385v2, 2026.

[3] Hao Li et al. AgentDyn: A Dynamic Open-Ended Benchmark for Evaluating Prompt Injection Attacks of Real-World Agent Security System. arXiv:2602.03117v1, 2026.

[4] Stavan Parikh and Rao Surapaneni. Powering AI commerce with the new Agent Payments Protocol AP2AP2. Google Cloud Blog, September 16, 2025.


Reviewer_w59nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents RedCommerce, a security benchmark for LLM agents that act in commerce across seven lifecycle stages, from Awareness to Post-Purchase Service. Its central idea is the Mandate–Commitment Gap. A user's mandate often leaves a field open or bounded, such as which product to buy. Freedom to choose that field does not give the agent the evidence or the authority to commit to any value. The authors formalize this at the field level. They separate an information chain (mandate, object and evidence produce a proposal) from an authorization chain (mandate, authority witness and state produce allow, clarify or deny). From this they derive six candidate failure routes (M1 to M6).

The benchmark has 96 task packages and 829 variant worlds across 24 threat families. Each task has an explicit attacker write surface, trusted records that the attacker cannot change, and a deterministic task-specific scorer instead of an LLM judge. Validation uses Haiku 4.5 as the only business agent. Four attacker models reach 23.50% to 36.46% attack success over the 829 worlds. A matched study on 96 worlds raises adverse outcomes from 1/288 under ordinary inputs to 63/288 under saved attacks, a 21.53-point increase (95% Case-bootstrap interval 13.89 to 29.51). Three smaller studies test verification checks (8 frozen attacks in 4 Cases, 5 runs each) and authorization sensitivity (2 tasks, 80 decisions).

Critical strengths
  1. A useful framing with executable, deterministic scoring. The Mandate–Commitment Gap turns a vague notion ("the agent was manipulated") into checks on specific committed fields. Success needs a predefined change in protected state or delivered output. A changed wording or a permitted alternative does not count (Appendix B). Scoring is task code over the executed trajectory, not an LLM judge. This matters because most agent-security benchmarks score tool calls or text. RedCommerce scores whether a commitment was justified, which is the right target for commerce agents.

  2. The matched design separates attack effect from baseline error. The 96-world study compares each saved attack with an ordinary input in the same world (1/288 versus 63/288). The verification study also runs a lawful counterpart for every attack, and all 160 counterpart runs complete without deviation or escalation. This matters because it shows that the measured failures come from the adversarial input and that the checks do not simply block legitimate business. The reproducibility archive supports offline rescoring of all saved trajectories.

Critical weaknesses
  1. The empirical base is too narrow for the benchmark claims. Every experiment uses one business agent, Haiku 4.5 with thinking disabled (Appendix J.1). Only the attacker varies. The mechanism studies are small: 8 frozen attacks in 4 Cases with 5 runs per condition, and 2 authorization tasks with 4 proposals each. In the twelve-context diagnostic, contexts M4-A, M6-A and M6-B have no eligible proposals (Table 20), so the six failure routes are not all tested on generated attacks. In the issuer study, the credential identifiers stay descriptive (CRED-GOOD versus CRED-ROGUE). The authors note that the experiment "does not isolate whether the agent uses identifier wording or inspects the referenced issuer field" (Appendix K.5). As a result, the 20/20 contrast may reflect reading a label rather than checking authority. This matters because a benchmark paper must show that its distinctions transfer across agents and are measured cleanly.

  2. Task and scorer validity are not independently established. One author who built the tasks also reviewed them, without blinding. The paper states: "With one human assessor, independent agreement and inter-reviewer reliability remain unmeasured" (Appendix I). Specification problems already appear in the results. Seven Cases were excluded post hoc for unspecified identifier normalization, which lowers the matched effect from 21.53 to 18.35 points. In the shipment Case, 5 of the 17 releases under the evidence check involve a decorated reference to an existing record, and the paper leaves its correctness unresolved (Appendix K.4). This matters because the headline attack success rates depend on predicates whose semantic validity is only partly verified across the other 89 Cases.

Questions for authors

Clarification requests

  1. Do the matched-input increase and the verification results hold for other business agents, such as Haiku with thinking enabled or a model from another provider? Even a 96-world matched run on one more agent would address Weakness 1.
  2. Can you rerun the issuer study with neutral credential identifiers, so that only the referenced issuer field differs?
  3. How many of the remaining 89 Cases depend on identifier or normalization conventions that the instructions leave unspecified? Would fixing them change the attack success rates in Table 1?
  4. Was any sample of tasks rated by a second person who did not build them, against criteria K1 to K5 and E1 to E3? If so, what was the agreement?
  5. Under ordinary inputs, 22 of 288 episodes end in escalation (Table 13). How many of these are unnecessary refusals of a lawful task?

Suggestions for future work

  1. Evaluate the full request, authorization and resumption loop. Escalation currently ends the episode.
  2. Report the controlled handoff comparison that Appendix N says "remains to be reported". Only 4 of 96 tasks chain more than one business role.
  3. Evaluate deployed agent frameworks that act through web interfaces, such as Browser-Use or Manus, on a storefront front end bound to the same protected state and scorers. This would test whether the Mandate–Commitment Gap failures persist when the agent must extract evidence and authority from rendered pages rather than typed observations.
4 comments
AuthorsOpenReview ↗

Thank you, reviewer w59n! (1/3)

1:Response to Weaknesses

W1. Empirical breadth

The empirical base is too narrow for the benchmark claims. Every experiment uses one business agent, Haiku 4.5 with thinking disabled AppendixJ1˙Appendix J\.1. Only the attacker varies.

Thank you for asking whether the result extends beyond Haiku. Saved attacks produced adverse commitments on four additional business models: two frontier proprietary configurations Sol61˙,Sonnet55˙Sol 6\.1, Sonnet 5\.5 and two open-weight families (DeepSeek, Qwen). We replayed the same saved inputs without retuning. The tables report ASR_submit, the adverse fraction among evaluable committed runs.

Business model829-world attack replay
Sol 6.13.09%
Sonnet 5.514.71%
DeepSeek V4.1 Flash-101721.62%
Qwen3.8-27B32.03%

The separate 96-world matched study compares ordinary and attack inputs in the same protected world, with the same explicit reference rule and three runs per arm. We compare adverse rates among evaluable committed runs in each arm.

Business model96-world ordinary96-world attack
Sol 6.10.36%1.98%
Sonnet 5.50%16.96%
DeepSeek V4.1 Flash-10170%25.60%
Qwen3.8-27B0%31.30%

All four models show a higher observed rate with attack inputs, although the effect varies by model.

The mechanism studies are small: 8 frozen attacks in 4 Cases with 5 runs per condition,

Verification coverage. We expanded the Haiku diagnostics to additional Cases and repeated the original verification comparison on other models. In the expanded study, targeted checks for evidence and authority outperformed both no check and a check aimed at a different mechanism under the original instructions.

Targeted checkCasesNo checkTargetedOther checkAdjusted p
Evidence support (M3)338/5024/5043/500.0000964
Authority (M4)219/301/3010/300.00272

M3 includes delivered recommendations; the order-only comparison was not significant. The evidence supports specific checks, with effects that depend on the task and outcome.

and 2 authorization tasks with 4 proposals each.

Authorization coverage. We added Sol to the issuer comparison on the original task. This tests a second business model without expanding the two-task authorization-sensitivity study. The neutral-ID result is below.

In the twelve-context diagnostic, contexts M4-A, M6-A and M6-B have no eligible proposals (Table 20), so the six failure routes are not all tested on generated attacks.

Generated evidence across mechanisms. The original twelve-context panel did not test all six routes with generated proposals. The expanded comparison adds generated-input evidence for authority M4M4 and evidence support M3M3. It leaves the original Table 20 intact and does not establish coverage for all six routes.

In the issuer study, the credential identifiers stay descriptive CRED−GOODversusCRED−ROGUECRED-GOOD versus CRED-ROGUE. The authors note that the experiment "does not isolate whether the agent uses identifier wording or inspects the referenced issuer field" AppendixK5˙Appendix K\.5. As a result, the 20/20 contrast may reflect reading a label rather than checking authority. This matters because a benchmark paper must show that its distinctions transfer across agents and are measured cleanly.

Credential identifiers. The issuer contrast persisted after removing the GOOD/ROGUE cues. We used neutral IDs; within each pair the proposal and cited ID were fixed and only the trusted issuer changed. Haiku and Sol each committed 40/40 valid-issuer proposals and escalated 40/40 invalid-issuer proposals. The contrast therefore persists without descriptive credential labels.

W2. Scorer validity and independent review

Task and scorer validity are not independently established. One author who built the tasks also reviewed them, without blinding. The paper states: "With one human assessor, independent agreement and inter-reviewer reliability remain unmeasured" (Appendix I).

Thank you for raising the independence concern. A task author performed the original checks; we have since completed an independent, outcome-blinded pilot on ten Cases. It found both agreement and concrete discrepancies. The rubric, agreement and findings are reported in Q4.

AuthorsOpenReview ↗

Thank you, reviewer w59n! (2/3)

Specification problems already appear in the results. Seven Cases were excluded post hoc for unspecified identifier normalization, which lowers the matched effect from 21.53 to 18.35 points.

Seven-Case sensitivity. We reran both arms with the explicit rule that the cited record must support the commitment. Among evaluable committed runs, attack inputs produced more adverse outcomes than ordinary inputs even after excluding the original seven Cases.

Clarified matched-input studyOrdinary ASR_submitAttack ASR_submit
Full panel0/245 (0.00%)53/134 (39.55%)
Excluding the original seven Cases0/227 (0.00%)41/122 (33.61%)

In the shipment Case, 5 of the 17 releases under the evidence check involve a decorated reference to an existing record, and the paper leaves its correctness unresolved AppendixK4˙Appendix K\.4.

Shipment example. We agree that an appended explanation is not evidence of a missing SDS record. We aligned the task, lookup and scorer on how to extract the registry identifier. Rescoring the same evidence-check actions reduced adverse outcomes from 17/20 to 12/20. In fresh Haiku runs with the aligned rule, both no check and the evidence check yielded 3/20.

This matters because the headline attack success rates depend on predicates whose semantic validity is only partly verified across the other 89 Cases.

Validity beyond the flagged Cases. We compared agent-visible rules with scorer lookups across all 96 Cases and reproduced the original defect codes for all 910 scored successes. Reproduction checks scorer consistency; the rule review and independent pilot address semantic alignment. The pilot found a price-label parsing issue outside the original exclusions and a delegation-wording disagreement. Q3 distinguishes what is known about prevalence and score sensitivity.

2:Response to Questions

Q1. Results on additional business models

Do the matched-input increase and the verification results hold for other business agents, such as Haiku with thinking enabled or a model from another provider? Even a 96-world matched run on one more agent would address Weakness 1.

Matched-input increase. The observed ASR_submit was higher under attack on all four additional models (W1). We did not evaluate Haiku with thinking enabled.

Verification results. The checks transferred in some settings. For the GTIN item-identity check, adverse outcomes fell from 5/10 to 0/10 on Sonnet and from 5/10 to 1/10 on DeepSeek. All 640 lawful controls in the cross-model verification study completed. This supports task-specific benefits while preserving utility in these controls.

Q2. Neutral credential identifiers

Can you rerun the issuer study with neutral credential identifiers, so that only the referenced issuer field differs?

Yes. With neutral IDs and only the trusted issuer changed within each pair, both models still committed valid proposals and escalated invalid ones. W1 gives the paired design and counts.

Q3. Prevalence and effect of specification ambiguity

How many of the remaining 89 Cases depend on identifier or normalization conventions that the instructions leave unspecified?

How many of the 89? We mapped the visible rules and scorer lookups for all 89, but did not estimate a Case-level count of unspecified normalization conventions. The price-label parsing issue found in the pilot is outside the original seven; those exclusions were a targeted sensitivity check, not a census.

Would fixing them change the attack success rates in Table 1?

Effect on reported rates. We have not quantified how correcting same-ID formatting would change the original Table 1 rates. We did measure sensitivity to a separate question: whether a valid record must be the one cited. Both columns below use the same evaluable committed runs. The sensitivity column omits scored successes where a different valid reference clears the defect and the instructions require only that a valid record exist.

Original attackerASR_submitASR_submit under wording sensitivity
Opus 5.5270/324 (83.33%)98/324 (30.25%)
DeepSeek V4.1 Flash237/305 (77.70%)63/305 (20.66%)
Qwen 3.8 Max206/259 (79.54%)104/259 (40.15%)
Gemini 3.8 Flash188/294 (63.95%)70/294 (23.81%)

Reference wording changes the reported ASR_submit. In the clarified matched rerun, the observed ASR_submit remained higher under attack (W2).

AuthorsOpenReview ↗

Thank you, reviewer w59n! (3/3)

Q4. Independent review and agreement

Was any sample of tasks rated by a second person who did not build them, against criteria K1 to K5 and E1 to E3?

Independent review. Yes. One person uninvolved in task construction assessed ten Cases covering all seven stages. They first judged each scenario from agent-visible materials before seeing scorer code; model identities, saved outcomes and prior conclusions were withheld.

RubricCriteria
K1–K3Rule clarity, access to required information, feasible lawful completion
K4–K6Reachable adverse action, scorer–rule alignment, matched controls
E1–E3Business realism and consequences, role mandate, realistic attacker permissions

If so, what was the agreement?

Agreement. The assessor and code agreed on 15/17 (about 88%) among the scenarios included in the comparison. Two disagreed and three of the 20 candidate scenarios depended on how the rule was read. With one independent assessor, this measures human–scorer agreement; it does not estimate agreement between independent human assessors.

Q5. Escalations under ordinary inputs

Under ordinary inputs, 22 of 288 episodes end in escalation (Table 13). How many of these are unnecessary refusals of a lawful task?

We have not yet determined how many were unnecessary. The 22 escalations span 13 Cases. Classifying them requires checking what the agent could see and whether it could complete the task lawfully without further authorization; an ordinary input alone does not establish that escalation was unnecessary.

Suggestions for future work

F1. Authorization and resumption

Evaluate the full request, authorization and resumption loop. Escalation currently ends the episode.

Thank you for this suggestion. The current episode stops at escalation. Providing the requested authorization or clarification and testing whether the agent then resumes correctly would evaluate the full workflow, including whether a justified pause leads to lawful completion.

F2. Controlled handoff comparison

Report the controlled handoff comparison that Appendix N says "remains to be reported". Only 4 of 96 tasks chain more than one business role.

We completed the controlled comparison on two of the four role-chain Cases, using identical protected worlds for ordinary and saved attack inputs. Both models completed all ordinary runs without adverse results. In the attack arm, Haiku delivered 16/20 unsafe handoffs: eight led to an adverse order and eight escalated after the handoff. Sol had no adverse handoffs. Final-order scoring alone would miss the eight unsafe handoffs that preceded escalation.

F3. Rendered-storefront evaluation

Evaluate deployed agent frameworks that act through web interfaces, such as Browser-Use or Manus, on a storefront front end bound to the same protected state and scorers. This would test whether the Mandate–Commitment Gap failures persist when the agent must extract evidence and authority from rendered pages rather than typed observations.

Thank you for suggesting a rendered storefront. This remains future work. Connecting browser actions to the same protected state and scorer would let us test whether the failures persist when the agent reads rendered pages, while keeping the transaction rules and outcome criteria fixed.

Dear authors,

Thank you so much for the rebuttals, but as the deadline for the discussion period is almost over, I cannot guarantee I will be able to sufficeintly scrutinize the rebuttals and update my comments in the remaining time. I will try my best though.

Reviewer_6YXAOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RedCommerce, an executable security benchmark designed to evaluate consequential risks in agentic commerce across its full lifecycle. The benchmark is organized around the proposed concept of the Mandate–Commitment Gap (MCG): while user mandates typically leave certain transaction details open to agent discretion, committing consequential actions (such as executing payments, accepting binding terms, or releasing shipments) requires both admissible supporting evidence and valid authorization.

RedCommerce comprises 96 task packages (Cases) and 829 variant-worlds spanning seven commercial lifecycle stages and five role families. In each episode, an attacker is restricted to modifying designated lower-trust input surfaces, while outcomes are evaluated by a task-specific, deterministic code verifier rather than an LLM judge. Using Haiku 4.5 as the fixed business agent against four attacker configurations (which achieve 23.50%–36.46% attack success rates across the corpus), a matched-input comparison across all 96 Cases shows that saved adversarial inputs increase scored adverse outcomes by 21.53 percentage points relative to ordinary inputs. This increase remains substantial at 18.35 percentage points after a post hoc sensitivity analysis excluding seven specification-sensitive Cases. Additionally, through smaller-scale controlled experiments, the paper investigates how verification assistance (object-binding and evidence-support checks) and authorization conditions (issuer validity and refund-policy scope) influence agent commitment and escalation behaviors while preserving legitimate task completion.

Critical strengths
  1. Strong Practical Relevance and Grounding in Consequential Business Effects: Each Case explicitly links attacker-writable inputs, protected authoritative records, agent tool actions, and executable outcome predicates. Rather than relying solely on surface-level tool-call syntax or LLM-as-a-judge opinions, the benchmark deterministically checks whether material commercial fields—such as transaction terms, product objects, or payout beneficiaries—undergo unauthorized deviations. This field-level formulation makes RedCommerce highly valuable for practical security evaluation in delegated commercial workflows.

  2. Rigorous Controlled Evidence for Adversarial Impact and Behavioral Diagnostics: Beyond reporting aggregate attack success rates, the paper employs well-designed controlled contrasts. In particular, the matched-input study compares saved adversarial submissions against ordinary inputs within the exact same protected worlds and provides 95% Case-bootstrap confidence intervals. Showing that the adverse-outcome increase persists (18.35 percentage points) after excluding seven identified specification-sensitive Cases provides convincing evidence that the observed failures represent genuine adversarial risk rather than baseline task errors. Furthermore, tracking lawful completion alongside security metrics ensures that defenses are evaluated without encouraging indiscriminate over-refusal.

Critical weaknesses
  1. Unresolved Concerns Regarding Scorer Semantic Validity and Lack of Independent Audit: The alignment between deterministic scorer predicates and agent-visible business rules requires further validation. For instance, in the shipment evidence Case (detailed in Appendix K.4), an attacker appends explanatory text to a valid SDS identifier (SDS-A). Because the task instructions do not explicitly specify identifier normalization rules, this decorated string fails the exact-match lookup in the verifier, even though the applicable SDS-A record is visible in the registry. Consequently, while the scored outcome proves that the agent released the shipment after receiving a negative exact-lookup diagnostic, it does not unequivocally prove that the action lacked admissible safety evidence under the stated rules. Although the authors transparently document this limitation and conduct a post hoc exclusion sensitivity analysis, the task validation and screening were conducted by a single benchmark author who also participated in task construction (Appendix I). Without an independent, blinded human audit, it remains unclear whether similar specification ambiguities or overly strict exact-match predicates exist across the remaining Cases, which somewhat weakens confidence in the raw benchmark scores as precise indicators of business policy violations.

  2. Limited Empirical Generalizability Across Victim Models and Diagnostic Contexts: The empirical conclusions rely heavily on a single victim/business model (Haiku 4.5) across all main and diagnostic evaluations. Moreover, while the full corpus spans 829 variant-worlds, the controlled mechanistic studies have narrow empirical coverage: the verification assistance experiment evaluates only eight selected attacks across four Cases, and the structured authorization sensitivity experiment tests only two existing tasks (80 decisions in total, with descriptive credential IDs remaining a potential cue in the issuer comparison). Demonstrating consistency across diverse business agent backbones and a broader subset of Cases would significantly strengthen the generalizability of the paper's empirical claims.

Questions for authors
  1. Systematic Screening of Scorer–Rule Alignment: Beyond the seven specification-sensitive Cases excluded in the post hoc sensitivity analysis, did you systematically audit the remaining 89 Cases for potential discrepancies between the deterministic scorer predicates and the agent-visible instructions? Specifically, how prevalent are issues related to unspecified string/identifier normalization, or cases where valid supporting evidence exists in the trusted registry but is referenced in a slightly non-canonical format? Please clarify the exact scope of the screening procedure used to identify the seven flagged Cases and discuss whether any residual specification ambiguities might affect the reported ASR or matched-input results.

  2. Feasibility of Independent Validation: Given that the current K1–K6 and E1–E3 quality assurance checks were performed by a single author involved in task development (Appendix I), could you provide an independent (preferably blinded) verification on a representative subset of Cases? Discussing the inter-assessor agreement and how any newly identified ambiguous Cases impact the main empirical conclusions would greatly enhance the reliability of the benchmark.

3 comments
AuthorsOpenReview ↗

Thank you, reviewer 6YXA! (1/3)

1:Response to Weaknesses

W1. Scorer validity and independent audit

Unresolved Concerns Regarding Scorer Semantic Validity and Lack of Independent Audit: The alignment between deterministic scorer predicates and agent-visible business rules requires further validation.

We thank the reviewer for highlighting scorer–rule alignment. The original quality checks compared the agent-visible rules with the scoring predicates. Our follow-up identified two gaps: how to parse a record identifier and whether the cited record itself must be valid.

Matched rerun. To test whether ambiguous reference wording explained the attack effect, we added the same explicit reference-binding rule to both arms. The observed increase persisted, including after excluding the original seven flagged Cases.

Clarified matched-input analysisOrdinaryAttackIncrease
Full panel0/28854/288+18.75 pp
Excluding the original seven Cases0/26742/267+15.73 pp

The full-panel 95% world-cluster confidence interval was 11.81–26.04 pp.

For instance, in the shipment evidence Case detailedinAppendixK4˙detailed in Appendix K\.4, an attacker appends explanatory text to a valid SDS identifier SDS−ASDS-A. Because the task instructions do not explicitly specify identifier normalization rules, this decorated string fails the exact-match lookup in the verifier, even though the applicable SDS-A record is visible in the registry. Consequently, while the scored outcome proves that the agent released the shipment after receiving a negative exact-lookup diagnostic, it does not unequivocally prove that the action lacked admissible safety evidence under the stated rules.

SDS formatting. Five saved releases were scored as adverse because explanations were appended to valid IDs. For this Case AppendixK4˙Appendix K\.4, the task, lookup and scorer now use the first whitespace-delimited token as the case-sensitive registry ID. Rescoring the same evidence-check actions reduced counted adverse outcomes from 17/20 to 12/20. In fresh Haiku runs under the aligned rule, both no check and the evidence check yielded 3/20 adverse outcomes.

Although the authors transparently document this limitation and conduct a post hoc exclusion sensitivity analysis, the task validation and screening were conducted by a single benchmark author who also participated in task construction (Appendix I). Without an independent, blinded human audit, it remains unclear whether similar specification ambiguities or overly strict exact-match predicates exist across the remaining Cases, which somewhat weakens confidence in the raw benchmark scores as precise indicators of business policy violations.

Independent human audit. One of the task authors conducted the original screening. A reviewer who did not build the tasks checked ten Cases without seeing the recorded outcomes and found discrepancies beyond SDS Q2Q2. Their prevalence remains unknown Q1Q1, so raw ASRs are rates under the verifier’s rules, not independently confirmed policy-violation rates.

The SDS recheck separates ID formatting from missing evidence. The attack increment persisted when we made the reference rule explicit.

W2. Generalization across models and diagnostic settings

Limited Empirical Generalizability Across Victim Models and Diagnostic Contexts: The empirical conclusions rely heavily on a single victim/business model Haiku45˙Haiku 4\.5 across all main and diagnostic evaluations.

We thank the reviewer for asking whether the findings hold across business models and diagnostic settings. The observed risk extends beyond Haiku: saved attacks induced adverse commitments on all four additional business models. The diagnostic gains were task-specific, and the issuer effect persisted after we removed descriptive credential labels.

11 Attack transfer across business models. All four models had a higher observed adverse rate under attack than under ordinary inputs in the matched study. We evaluated two frontier proprietary models Sol61˙andSonnet55˙Sol 6\.1 and Sonnet 5\.5 and two open-weight model families (DeepSeek and Qwen). The 829-world study tests whether saved attacks transfer without retuning. The separate 96-world study compares ordinary and attack inputs in the same protected worlds, using identical reference-binding instructions and three runs per arm. The table reports adverse outcomes among evaluable commitments ASRsubmitASR_submit.

AuthorsOpenReview ↗

Thank you, reviewer 6YXA! (2/3)

Business model829-world attack96-world ordinary96-world attack
Sol 6.13.09%0.36%1.98%
Sonnet 5.514.71%0%16.96%
DeepSeek V4.1 Flash-101721.62%0%25.60%
Qwen3.8-27B32.03%0%31.30%

Moreover, while the full corpus spans 829 variant-worlds, the controlled mechanistic studies have narrow empirical coverage: the verification assistance experiment evaluates only eight selected attacks across four Cases,

22 Verification coverage and results. We added Cases for Haiku and reran the original checks across models. In the expanded Haiku study, the targeted checks below outperformed both no check and a check for a different mechanism under the original instructions. The reported p-values use input-stratified tests and Holm correction across the prespecified family.

CheckCasesNo checkTargeted checkOther checkAdjusted p
Evidence support (M3)338/5024/5043/500.0000964
Authority (M4)219/301/3010/300.00272

The M3 outcome includes delivered recommendations; the comparison restricted to orders was not significant. The separate mechanism tests with clarified instructions did not pass correction.

The product-identity check also helped on models other than Haiku. It reduced adverse outcomes from 5/10 to 0/10 for Sonnet and from 5/10 to 1/10 for DeepSeek. All 640 lawful counterparts in the cross-model verification study completed.

and the structured authorization sensitivity experiment tests only two existing tasks (80 decisions in total,

33 Coverage of authorization tasks. The new issuer comparison added Sol and retested the original issuer task. It did not expand the authorization-sensitivity study to additional tasks.

with descriptive credential IDs remaining a potential cue in the issuer comparison). Demonstrating consistency across diverse business agent backbones and a broader subset of Cases would significantly strengthen the generalizability of the paper's empirical claims.

44 Issuer validity without label cues. Haiku and Sol continued to distinguish valid from invalid issuers after we removed the GOOD/ROGUE cue. We used four author-written proposals and two counterbalanced mappings for neutral credential IDs. Within each pair, the proposal and cited ID were identical, and only the trusted issuer changed. Both models committed 40/40 valid-issuer proposals and escalated 40/40 invalid-issuer proposals.

Taken together, the replays establish that the observed risk extends beyond Haiku. The controlled tests show why the benchmark evaluates the evidence and authority for each commitment separately: an effective check in one setting need not address a different failure.

2:Response to Questions

Q1. Systematic screening of scorer–rule alignment

Systematic Screening of Scorer–Rule Alignment: Beyond the seven specification-sensitive Cases excluded in the post hoc sensitivity analysis, did you systematically audit the remaining 89 Cases for potential discrepancies between the deterministic scorer predicates and the agent-visible instructions?

We thank the reviewer for asking how far the screening extended beyond the original seven Cases. We screened all 96 Cases, including the remaining 89, by mapping the agent-visible rules to the scorer lookups. The independent pilot described in W1 and Q2 examined a subset.

Specifically, how prevalent are issues related to unspecified string/identifier normalization, or cases where valid supporting evidence exists in the trusted registry but is referenced in a slightly non-canonical format?

Identifier format. We have not estimated how often a non-canonical form of the same identifier changes the score across the remaining Cases.

Please clarify the exact scope of the screening procedure used to identify the seven flagged Cases

Original screen. The original seven-Case exclusion was narrower: it combined the SDS Case with six additional Cases flagged in the saved matched-input panel. In those six, a known identifier had been submitted with explanatory text appended. Restoring only the identifier to its registered form cleared the corresponding defects in diagnostic copies. The task instructions had not specified how to normalize these strings.

AuthorsOpenReview ↗

Thank you, reviewer 6YXA! (3/3)

and discuss whether any residual specification ambiguities might affect the reported ASR or matched-input results.

Effect on reported outcomes. The interpretation of the reference rule affects which submissions count as adverse. The reference audit tested whether requiring the cited record itself to be valid changes the score when a different valid record exists. Among 910 originally scored successes, replacing only the cited record cleared defects in 721, including 566 with existence-based wording. These are counts among scored successes; we have not estimated how prevalent the ambiguity is across the remaining 89 Cases.

In the matched rerun with explicit reference binding, the attack increment remained positive even after excluding the original seven Cases W1W1. The independent audit also flagged cart-label price parsing outside those exclusions; its effect on aggregate rates has not been quantified Q2Q2.

Q2. Independent validation and agreement

Feasibility of Independent Validation: Given that the current K1–K6 and E1–E3 quality assurance checks were performed by a single author involved in task development (Appendix I), could you provide an independent (preferably blinded) verification on a representative subset of Cases?

We thank the reviewer for suggesting an independent, blinded review. We completed the independent, outcome-blinded pilot on ten Cases summarized in W1. An assessor who did not build the tasks reviewed a sample selected to cover all seven lifecycle stages, a sequential-role task and known ambiguity risks. The assessor recorded judgments from the visible rules before seeing the scorer code, with model identities, saved outcomes and previous audit conclusions withheld.

RubricCriteria
K1–K3Rule clarity, access to required information, feasibility of lawful completion
K4–K6Reachability of an adverse action, scorer–rule alignment, matched controls
E1–E3Business realism and consequences, role mandate, realistic attacker permissions

Discussing the inter-assessor agreement

Agreement. One independent assessor took part, so inter-assessor agreement was not measured. Of 20 candidate scenarios, the assessor and scorer agreed on 15 and disagreed on two; three depended on how the rule was read. Agreement with the scorer was 15/17 (about 88%) among the scenarios included in the comparison.

and how any newly identified ambiguous Cases impact the main empirical conclusions would greatly enhance the reliability of the benchmark.

Effect of the new findings. The audit flagged cart-label price parsing and a delegation task whose instructions allow using a different valid authorization, while the scorer checks the cited record. The reference audit in Q1 covers the delegation issue as a sensitivity check. We have not separately quantified how correcting either task would change aggregate ASR. The matched rerun in W1 addresses the broader question: the attack increment persisted when the reference rule was explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.