acceptodds
Under review as a conference paper at ICLR 2027

When Shopping Agents Are Misled: Understanding and Mitigating Purchase Safety Failures under Deceptive Merchants

Abstract

Shopping agents are increasingly asked to make purchase decisions, but existing e-commerce benchmarks primarily evaluate task completion in benign shopping environments. However, real-world shopping often involves deceptive merchants whose evidence and dialogue can lead agents to unsafe purchases. To bridge this gap, we introduce SafeShopBench, a purchase-safety benchmark for evaluating shopping agents in adversarial merchant environments. Experimental results demonstrate that SafeShopBench poses a substantial challenge even for frontier shopping agents. In our full-tool evaluation, unsafe-purchase rates across the twelve buyer agents range from 14.3% to 51.8%, and even the best-performing agent on this metric, Claude Opus 5, makes unsafe purchases in 14.3% of adversarial cases, highlighting the difficulty of purchase safety under deceptive merchants. To mitigate these purchase-safety failures, we evaluate two complementary approaches: a tool-grounded Detector-Corrector wrapper that revisits purchase decisions without retraining the buyer, and supervised fine-tuning followed by outcome-based reinforcement learning. For a smaller buyer agent, reinforcement learning reduces the unsafe-purchase rate from 24.9% after supervised fine-tuning to 8.3%, below that of Claude Opus 5. These results suggest that SafeShopBench supports both inference-time correction and the training of safer shopping agents.

Primary area:
datasets and benchmarks
Keywords:
shopping agents, purchase safety, deceptive merchants, agentic commerce

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ngN2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates purchase safety for shopping agents when facing deceptive merchants. The authors introduce SafeShopBench, a benchmark containing 1,200 paired cases across 5 product categories and 12 merchant-deception scenarios. Each case includes a normal Reference Setting and an Adversarial Setting with an additional deceptive unsafe product, allowing the authors to evaluate whether merchant deception leads agents to make unsafe purchase decisions.

The authors evaluate 12 shopping agents and report unsafe-purchase rates ranging from 14.3% to 51.8% in the adversarial setting. They also use the Unsafe-Flip Rate to analyze how previously correct purchase decisions turn into unsafe purchases. In addition, the authors investigate two mitigation approaches: Detector–Corrector, which requires no retraining, and SFT + Decision-GRPO.

Overall, the paper addresses a practically relevant problem, and the benchmark is relatively comprehensive in its construction.

Critical strengths

S1. The research problem is practically relevant and well motivated.

As shopping agents become increasingly involved in making purchase decisions, misleading information from merchants may cause them to violate users' actual shopping requirements. The paper distinguishes this issue from traditional shopping task completion, providing a clear motivation for the study.

S2. The benchmark is systematically constructed, with substantial experimental effort.

The authors construct 1,200 paired cases covering different product categories and various merchant-deception scenarios. They also design tools for retrieving information from multiple sources and introduce data quality checks. In addition, the paper evaluates 12 different models and provides scenario-level analyses and experiments on two mitigation approaches. Overall, the experimental coverage is fairly extensive.

Critical weaknesses

W1. A substantial number of reverse decision changes in the paired evaluation raise concerns about the benchmark's reliability.

The Adversarial Setting preserves the original shopping instruction and candidate products from the Reference Setting, adding only one deceptive unsafe product. Under this design, additional interference would generally be expected to make the shopping task more difficult. However, the results seem to suggest that many previously incorrect purchase decisions become correct in the adversarial setting. For example, in Table 2, the correct-purchase rate of Claude Opus 5 decreases only slightly from 65.3% to 65.1%, while its Unsafe-Flip Rate reaches 13.6%, implying a considerable number of changes in the opposite direction. Although agent decisions are not necessarily strictly monotonic, this observation raises concerns about whether other factors in the benchmark construction, interactive environment, or evaluation process may be affecting the results. The paper does not explain this phenomenon, which weakens confidence in the benchmark and the reliability of its paired evaluation results.

W2. The experiments do not sufficiently distinguish the effects of merchant deception from other factors affecting purchase decisions.

The Reference Setting contains 9 candidate products, while the Adversarial Setting contains 10. This means that the two settings differ not only in deceptive information but also in the number of candidate products. As a result, it is difficult to determine whether the observed decrease in purchase success rates is caused by the increased number of candidates or by merchant deception itself.

W3. The technical contributions and experimental evidence for the two mitigation approaches are relatively limited.

Detector–Corrector essentially adds an additional verification and correction step after the original purchase decision. Meanwhile, Decision-GRPO uses the final purchase outcome as the reinforcement learning reward, but the main text does not sufficiently clarify its technical differences from existing outcome-reward-based reinforcement learning methods. Therefore, these approaches appear more like applications of existing techniques to the new benchmark rather than methods with substantial technical novelty.

The authors' additional experiments and clarifications have addressed most of my concerns, and I will raise my score from 2 to 3 accordingly.

Questions for authors

Q1. Reverse decision changes in the paired evaluation

Why do a considerable number of incorrect purchase decisions in the Reference Setting become correct in the Adversarial Setting when only one deceptive product is added and the other original conditions remain unchanged? In particular, how do the authors explain the results for Claude Opus 5?

Q2. Implementation details of the shopping agents

Do all Buyer agents use exactly the same agent framework, with only the underlying language model replaced? Are the system prompts, tool-calling procedures, and runtime parameters consistent across different models?

C1. A more complete analysis of paired evaluation results

I suggest that the authors provide a complete Reference–Adversarial outcome transition matrix. This should cover not only cases where correct purchases become unsafe purchases, but also cases where previously incorrect purchase decisions become correct. The authors should further investigate whether these changes are related to the benchmark construction or evaluation process.

7 comments
AuthorsOpenReview ↗

Response to Reviewer ngN2 (Part 1/5)

We appreciate your careful examination of the experimental design, the interpretation of paired outcomes, and the technical scope of the mitigation approaches. Your questions help us make these distinctions more explicit; below, we address reverse transitions, benchmark and evaluation checks, candidate-set size, the shared agent protocol, and the role of the mitigation methods.

AuthorsOpenReview ↗

Response to Reviewer ngN2 (Part 2/5)

W1 / Q1: Why do a considerable number of incorrect purchase decisions in the Reference Setting become correct in the Adversarial Setting when only one deceptive product is added and the other original conditions remain unchanged? In particular, how do the authors explain the results for Claude Opus 5?

A1: Thank you for highlighting these reverse transitions. Our trajectory analysis identifies a recurring pattern: the adversarial run changes which products the agent compares or which user requirements it prioritizes, allowing it to select the correct product after making an incorrect reference decision.

Cross-model trajectory analysis. We analyzed available reference–adversarial trajectories for the three proprietary Buyers with the largest non-correct-to-correct transition counts: MiMo-V2.5 (95), MiMo-V2.5-Pro (93), and Claude Opus 5 (90), and the top two open-source Buyers, GLM-4.7-Flash (98) and Qwen3.5-4B (81).

Illustrative cases. We selected the following two cases from this analysis:

  • Qwen3.5-4B: changing the comparison set. The user requested Rhode Pocket Blush in stick form. In the reference run, the agent compared A/J and selected the cheaper cream-form product J for its price and favorable seller and return terms, overlooking the requested stick format supported by A. In the adversarial run, it instead compared A with the added deceptive candidate H. After identifying H’s lack of manufacturer authorization and seller-only warranty, it selected A, which matched the requested product form. The comparison shifted from J’s price and seller benefits to H’s authorization and warranty problems, leading the agent to select the matching product A.
  • Claude Opus 5: shifting from price advantages to a better-supported match. The user requested breathable, wear-resistant, full-finger touchscreen gloves. In the reference run, the agent compared E/H and selected E for its lower price and better return terms, overlooking the need to verify the breathability and wear-resistance requirements explicitly documented for H. In the adversarial run, it instead compared the added deceptive candidate F against H. After accounting for F’s mandatory surcharge, it selected H, which explicitly supported the requested attributes. The comparison shifted from E’s price and return advantages to F’s surcharge, leading the agent to select the product supported by the requested specifications.

Implications for paired evaluation. An agent may favor a benign but non-target product over the gold product in the reference run, yet compare the gold product against the added deceptive candidate in the adversarial run and select the gold product. Such reverse transitions coexist with more frequent decision degradation: correct-to-unsafe transitions outnumber reverse transitions for all 12 Buyers. Aggregate accuracy reflects only the net change; paired outcomes distinguish correct decisions that become unsafe from initially non-correct decisions that become correct. The cross-model transition table in our response to C1 makes both directions explicit.

Claude-specific clarification. Our closer inspection also identified failures at the API execution stage in Claude's reference runs, where some tasks never entered the shopping workflow. We reran those reference tasks and incorporated 386 replacement runs, while retaining the archived adversarial records for this comparison. The corresponding main-table results are:

Table R5. Claude Opus 5 results with corrected reference runs.

BuyerReference CorrectAdversarial CorrectAdversarial UnsafeCorrect-rate dropUFR
Claude Opus 572.8% (873/1,200)65.2% (782/1,200)15.1% (181/1,200)7.6 pp13.7% (120/873)

Claude’s results thus involve two distinct issues. The original reference-side execution failures affected its aggregate correctness statistics and required correction. Separately, the corrected comparison contains 90 non-correct-to-correct transitions, comprising 86 incorrect purchases and four rejections that became correct purchases. These coexist with 120 correct-to-unsafe transitions. Overall, correct outcomes decrease from 873 to 782, a net reduction of 91.

The gloves case illustrates a recurring pattern in the Claude reverse-transition trajectories we inspected: a shift in the alternatives being compared. As detailed above, Claude moved from favoring E over H on price and return terms to comparing F against H and selecting H after identifying F’s mandatory surcharge. It thus reaches the matching product through a different comparison, rather than because the original product information has changed.

AuthorsOpenReview ↗

Response to Reviewer ngN2 (Part 3/5)

C1: I suggest that the authors provide a complete Reference–Adversarial outcome transition matrix. This should cover not only cases where correct purchases become unsafe purchases, but also cases where previously incorrect purchase decisions become correct. The authors should further investigate whether these changes are related to the benchmark construction or evaluation process.

A2: Thank you for suggesting a bidirectional analysis. We report both correct-to-unsafe and non-correct-to-correct transitions across all 12 Buyers, using all 1,200 paired cases per Buyer, and examine benchmark-construction and execution factors below.

Bidirectional transition results.

Table R6. Reference-to-adversarial outcome transition counts across 12 Buyers.

BuyerCorrect → UnsafeNon-correct → Correct
DeepSeek-V4-Flash25045
DeepSeek-V4-Pro29856
GPT-5.6 Sol17487
Claude Opus 512090
MiMo-V2.524195
MiMo-V2.5-Pro22393
Gemma-4-31B-it36819
GLM-4.7-Flash31598
Qwen3-4B42837
Qwen3.5-4B30681
Qwen3.5-9B24270
Qwen3.6-27B22050

All 12 Buyers exhibit more correct-to-unsafe transitions than reverse transitions. For Claude, 120 reference-correct purchases become unsafe, while 90 initially non-correct outcomes become correct. The paired counts distinguish safety failures from reverse transitions, providing information beyond the net decline in correctness.

Benchmark-construction checks. To investigate whether reverse transitions arose from changes to the original product information, we compared the shared product details in the glove case discussed in W1/Q1. The relevant details were identical across settings, but Claude changed its comparison from E/H to F/H and selected H instead of E. In this case, the original product information remained unchanged; the reverse transition reflected a change in which alternatives the agent compared and ultimately selected.

Evaluation-process checks. Our examination of the evaluation records also identified Claude reference tasks that failed before entering the shopping workflow. We reran those tasks and incorporated the corrected reference results into the table, while retaining the archived adversarial records, as detailed in W1/Q1.

Reviewer_7ir9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SafeShopBench, a benchmark for evaluating the purchase safety of shopping agents in deceptive merchant environments. It uses paired shopping scenarios covering different merchant deception strategies to assess how such attacks affect purchasing decisions. The authors evaluate several LLM-based buyer agents and investigate two safety improvement approaches: a Detector–Corrector framework and SFT followed by RL. Experimental results reveal safety vulnerabilities in existing agents and show that both approaches can reduce unsafe purchases.

Critical strengths
  1. The paper is well-motivated and presents a well-designed evaluation framework. Its focus on purchase safety addresses an important gap in existing shopping agent benchmarks. In particular, the paired evaluation design provides a controlled way to measure safety failures caused by deceptive merchant offers, making the evaluation both reasonable and meaningful.

  2. The experiments are comprehensive and well-organized. The paper evaluates a diverse set of LLM-based buyer agents on SafeShopBench, providing detailed performance comparisons and analyses across different attack surfaces.

Critical weaknesses
  1. Limited strategy-level analysis of merchant deception. Although the paper reports the Unsafe Flip Rate across six attack surfaces, SafeShopBench encompasses twelve distinct merchant deception strategies. The current analysis does not provide a breakdown by individual deception strategy, making it difficult to assess their relative effectiveness and identify which strategies pose the greatest challenges to purchase safety.

  2. Although the proposed wrapper reduces the Unsafe Purchase Rate by 3-7 percentage points, it introduces approximately 53-57 seconds of additional latency and 9.9K-12.5K extra tokens per case on average, even after accounting for selective triggering. The absence of a cost-effectiveness comparison with more lightweight safety intervention approaches makes it difficult to assess the practical efficiency of this framework.

Questions for authors
  1. Figure 3(b) compares the Unsafe Flip Rate across six attack surfaces, while SafeShopBench covers twelve distinct merchant deception strategies. It would be helpful to include a more fine-grained breakdown of the Unsafe Flip Rate or Unsafe Purchase Rate for each deception strategy across different buyer agents.

  2. Does the full Detector–Corrector framework truly offer more value than a simple pre-purchase safety check? I suggest adding a low-cost safety intervention baseline, such as a brief risk check before the final purchase, rather than introducing the full Detector–Corrector framework.

  3. In Table 16, the selected checkpoints for the three RL training seeds correspond to different training steps (135, 360, and 360). Although the checkpoint selection procedure is explained, a brief discussion of why the selected steps differ would help clarify the training stability.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7ir9(Part 1/3)

Thank you for your careful review and concrete suggestions on strategy-level analysis, the pre-purchase safety-check baseline, and checkpoint selection. These comments help us explain the differences across deception strategies, the benefits and costs of the interventions, and the relationship between checkpoint selection and training stability.

W1 / Q1: Figure 3(b) compares the Unsafe Flip Rate across six attack surfaces, while SafeShopBench covers twelve distinct merchant deception strategies. It would be helpful to include a more fine-grained breakdown of the Unsafe Flip Rate or Unsafe Purchase Rate for each deception strategy across different buyer agents.

A1: Thank you for this suggestion. We provide the requested 12-strategy breakdown across all 12 buyer agents below. Following the UFR definition in Appendix B (p. 18), each entry is the percentage of reference-correct cases that become unsafe purchases under the corresponding strategy. Each strategy contains 100 cases; the denominator is the buyer's reference-correct subset.

Table R2a. Strategy-level unsafe-flip rates (%) for proprietary Buyers.

StrategyDS-FlashDS-ProGPT-5.6ClaudeMiMoMiMo-Pro
Prompt injection2.92.71.70.07.512.5
False product claim11.45.410.24.234.423.2
Information omission1.42.75.90.08.36.5
Identity spoofing61.583.344.88.060.771.2
Fake reputation41.167.150.80.063.262.3
Provenance confusion61.667.141.10.047.350.0
Fee manipulation8.33.915.50.012.110.5
After-sales obstruction24.329.18.61.423.015.5
Quote-state change62.761.985.741.360.957.6
Urgency-based social engineering30.023.015.11.430.237.5
Off-platform redirection11.38.53.60.013.013.3
Multi-turn persuasion33.833.810.20.052.557.1

Note. DS denotes DeepSeek-V4; GPT denotes GPT-5.6 Sol; Claude denotes Claude Opus 5; MiMo denotes MiMo-V2.5 (and Pro). Bold indicates the highest UFR in each buyer column.

Table R2b. Strategy-level unsafe-flip rates (%) for open-source Buyers.

StrategyGemmaGLMQ3-4BQ3.5-4BQ3.5-9BQ3.6-27B
Prompt injection10.350.089.036.910.218.0
False product claim14.727.937.915.218.212.5
Information omission4.310.322.41.60.01.6
Identity spoofing96.959.073.070.272.678.0
Fake reputation92.860.071.972.772.174.2
Provenance confusion91.055.273.969.269.542.9
Fee manipulation48.637.148.543.54.58.1
After-sales obstruction36.530.438.319.012.914.8
Quote-state change73.255.250.758.934.957.8
Urgency-based social engineering25.946.652.929.127.325.0
Off-platform redirection13.631.740.319.015.35.8
Multi-turn persuasion43.154.148.348.462.723.7

Note. Gemma denotes Gemma-4-31B-it; GLM denotes GLM-4.7-Flash; Q denotes Qwen. Bold indicates the highest UFR in each buyer column.

Two patterns emerge from this strategy-level breakdown. First, quote-state change is a shared vulnerability, with UFR ranging from 34.9% to 85.7% across buyers. This suggests that preserving purchase safety when transaction terms change remains challenging. Second, the most effective strategy depends on the buyer: among the evaluated strategies, prompt injection has the highest UFR for Qwen3-4B (89.0%), but only 1.7% for GPT-5.6 Sol, whose highest UFR occurs under quote-state change (85.7%).

AuthorsOpenReview ↗

Response to Reviewer 7ir9(Part 2/3)

W2 / Q2: Does the full Detector–Corrector framework truly offer more value than a simple pre-purchase safety check? I suggest adding a low-cost safety intervention baseline, such as a brief risk check before the final purchase, rather than introducing the full Detector–Corrector framework.

A2: Thank you for suggesting this practical baseline. Across all three evaluated Buyers, the full Detector–Corrector (D–C) framework achieves higher correct-purchase rates and lower unsafe-purchase rates than a single-pass pre-purchase safety check, at additional token cost. We evaluated the brief check on the same 1,200 adversarial cases per Buyer.

Experimental setup. The original Buyer performs one brief risk check before the final purchase. It can retain its decision, select another observed candidate, or reject all candidates, without additional tool calls. We use a fixed checking prompt across Buyers, with temperature 0 and a maximum of 512 output tokens.

Table R3. Purchase outcomes and additional token costs of test-time interventions.

BuyerMethodCorrect (%) ↑Unsafe (%) ↓Extra tokens/case ↓
Qwen3.6-27BNo intervention36.2526.250
Qwen3.6-27BSelf-check35.5826.427,515
Qwen3.6-27BD–C38.5822.0810,231
Gemma-4-31B-itNo intervention32.6739.080
Gemma-4-31B-itSelf-check32.4239.006,096
Gemma-4-31B-itD–C35.8333.259,857
GLM-4.7-FlashNo intervention28.9240.000
GLM-4.7-FlashSelf-check28.6739.506,320
GLM-4.7-FlashD–C33.0833.6712,542

Note. All rates use the full 1,200-case denominator. Token figures represent additional intervention cost per case, excluding the original shopping execution; the D–C values are trigger-weighted estimates reported in Appendix D.4 (p. 21).

Safety and task completion. Relative to self-check, D–C improves correct-purchase rates by 3.00–4.42 percentage points and reduces unsafe-purchase rates by 4.33–5.83 points. The brief check produces only small, mixed changes in unsafe purchases and slightly reduces correctness for all three Buyers. Across Buyers, it repairs five initially non-correct decisions but changes 19 initially correct decisions to non-correct outcomes, including 10 rejections. Thus, the simpler intervention costs less but does not match D–C in either correctness or safety in this evaluation.

Cost–benefit trade-off. D–C requires approximately 2.7K–6.2K more tokens per case than self-check. Its advantage in purchase safety and task completion therefore comes at additional token cost. The original D–C latency estimates are reported in Appendix D.4. Because self-check was measured on A800 hardware and D–C on A6000 hardware, these measurements do not provide a controlled latency comparison.

AuthorsOpenReview ↗

Response to Reviewer 7ir9(Part 3/3)

Q3: In Table 16, the selected checkpoints for the three RL training seeds correspond to different training steps (135, 360, and 360). Although the checkpoint selection procedure is explained, a brief discussion of why the selected steps differ would help clarify the training stability.

A3: Thank you for highlighting this point. The selected steps differ because development performance fluctuates across checkpoints, with narrow margins between the best candidates for two seeds. All three runs completed 360 training steps; step 135 denotes the selected checkpoint, not an earlier training termination.

Selection criterion. As described in Appendix E (pp. 23–24), we select among eight checkpoints using the lowest pooled adversarial unsafe-purchase rate on 120 held-out cases under two candidate orderings, yielding 240 case–ordering evaluations. These are two orderings of the same cases, and test-set outcomes are not used for selection. The best and second-best checkpoints are:

Table R4. Development-set performance of the selected and second-best checkpoints.

Training seedSelected stepDevelopment unsafeSecond-best stepDevelopment unsafeDifference in unsafe decisions
2026071313519/240 (7.92%)36020/240 (8.33%)1
2026071436019/240 (7.92%)31520/240 (8.33%)1
2026071536017/240 (7.08%)22522/240 (9.17%)5

Interpretation of training stability. The development curves decrease early in training but fluctuate thereafter, so their minima occur at different checkpoints. A large difference in selected steps does not necessarily imply a large difference in performance. For seed 20260713, step 360 produces only one more unsafe decision than step 135; its curve does not show sustained deterioration after the selected checkpoint. The best two checkpoints for seed 20260714 also differ by one decision, whereas seed 20260715 achieves its minimum at step 360 with five fewer unsafe decisions than its runner-up. All minima are unique, so the tie-breaking rule was not invoked.

The small cross-seed variation in Table 16 (sample SDs of 0.50 and 0.64 percentage points for adversarial correct- and unsafe-purchase rates) supports the consistency of the selected policies across these three runs: checkpoint selection varies across seeds, while the selected policies achieve similar final evaluation performance.

Reviewer_1uvDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The discussed topic of this paper is to build a shopping-agent benchmark only on the decision safety. That is, if the agent can tell the deceptive misleading caused by the merchants. The benchmark is based on pair-wise comparison, and the authors include several sota models into the evaluation. Moreover, they also showed a post-training method and a test-time method based on this benchmark to improve the models.

Critical strengths
  1. The discussed topic is critical in practical daily usage, and it meets the requirements of many big e-commerce companies.
Critical weaknesses
  1. I am a little doubted at the pair-wise comparison. While this is easy to build, it lacks the general standard which point-wise can bring to us. For example, in text-to-image reward models which I am more an expert in, most of them are point-wise based. If the authors can analysis or explain this pair-wise and point-wise design in the context of shopping agents (e.g., by related works), it will be helpful for my review.

  2. The proposed two methods (one test-time, one grpo-based) show a good experiment, however there is no baseline to compare them. Can authors provide some?

Questions for authors

See weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer 1uvD(Part 1/2)

We appreciate your careful assessment and constructive suggestions on evaluation design and baseline comparisons. Your comments help us clarify the complementary roles of independent outcome assessment and paired safety analysis, and make the comparisons supporting the mitigation approaches more explicit.


W1: I am a little doubted at the pair-wise comparison. While this is easy to build, it lacks the general standard which point-wise can bring to us. For example, in text-to-image reward models which I am more an expert in, most of them are point-wise based. If authors can analysis or explain this pair-wise and point-wise design in the context of shopping agents (e.g., by related works), it will be helpful for my review.

A1: We use pointwise evaluation to assess each shopping outcome independently. We add paired analysis to examine how outcomes change between reference and adversarial runs of the same task.

Independent outcome assessment. For pointwise evaluation, each scorable terminal decision is labeled according to predefined task requirements and purchase-safety criteria, independently of the counterpart run (Secs. 3 and 4.4 of the original submission). Table 2 reports absolute correct- and unsafe-purchase rates, with their definitions in Appendix B. This follows the task-based evaluation principle of WebShop [1] and ShoppingBench [2], which assess outcomes against product requirements and intent-specific constraints, respectively.

Paired analysis of shopping-agent safety. Paired analysis addresses an additional question about shopping-agent safety: does a task completed correctly in the reference setting end in an unsafe purchase under merchant deception? The unsafe-purchase rate measures how often unsafe purchases occur in the adversarial setting. The unsafe-flip rate measures how often reference-correct cases become unsafe (Sec. 3). Aggregate outcome rates alone cannot identify these transitions; matching runs of the same task makes them observable. This joint consideration of task performance and adversarial outcomes is consistent with AgentDojo [3], which reports benign utility, utility under attack, and targeted attack success.


References

[1] Shunyu Yao et al. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS, 2022.

[2] Jiangyuan Wang et al. ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents. AAAI, 2026.

[3] Edoardo Debenedetti et al. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS, 2024.

AuthorsOpenReview ↗

Response to Reviewer 1uvD(Part 2/2)

W2: The proposed two methods (one test-time, one grpo-based) show a good experiment, however there is no baseline to compare them. Can authors provide some?

A2: Thank you for suggesting explicit baseline comparisons. We have added a lightweight safety-check baseline for the test-time method; for Decision-GRPO, the original submission includes its SFT starting policy as a baseline in Table 4.

Test-time comparison. The low-cost baseline performs a brief risk check before the final purchase, without using the full Detector–Corrector (D–C) framework. We evaluated it on the same full set of 1,200 adversarial cases per Buyer. Each check uses the original Buyer model, which can retain its decision, select another observed candidate, or reject all candidates.

Table R1. Purchase outcomes under no intervention, self-check, and Detector–Corrector.

BuyerNo intervention: Correct (%)No intervention: Unsafe (%)Self-check: Correct (%)Self-check: Unsafe (%)D–C: Correct (%)D–C: Unsafe (%)
Qwen3.6-27B36.2526.2535.5826.4238.5822.08
Gemma-4-31B-it32.6739.0832.4239.0035.8333.25
GLM-4.7-Flash28.9240.0028.6739.5033.0833.67

The full D–C framework achieves higher correct-purchase rates and lower unsafe-purchase rates than the single-pass self-check across all three Buyers. The simpler intervention slightly reduces correctness, partly through conservative rejection: of the 19 previously correct decisions that became non-correct across the three Buyers, 10 became rejections, while only five previously non-correct decisions were repaired. Its effect on unsafe purchases is mixed, with small decreases for Gemma and GLM and a small increase for Qwen.

Training comparison. Decision-GRPO starts from the SFT policy, making SFT its direct baseline. Under the same evaluation protocol and on the same 1,200 paired tasks, Table 4 shows that adding Decision-GRPO increases adversarial correct purchases from 58.3% to 77.2% and reduces unsafe purchases from 24.9% to 8.3%. This comparison demonstrates the benefit of the additional Decision-GRPO training stage over SFT alone.

Reviewer_1uvDOpenReview ↗

Reply

I thank the authors for their effort. I am 70% satisfied with the rebuttal. Only 2 concerns: (1) For W1, I am doubted about the reliability of pair-wise check. Unlike other task, this shopping agent safety task is highly challenging and sometimes easy to mislead, therefore the pair-wise check can not be guaranteed as long as the single evaluation can not be highly guaranteed (I believe this requires a strict standard for single evaluation). (2) The GRPO vs. SFT is not sounded, because if you continue to SFT the base policy, there is a chance that it outperformes the GRPO (we have seen so many of this in LLM math reasoning), especially when data curation is available.

Overall, I will keep my score as weak rejection, with a low confidence as I do not have direct research experience on shopping agents.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful evaluations and constructive suggestions. Below, we summarize the revisions made in response to their feedback.

  1. Table 2 (Page 6): We updated Claude’s results after rerunning reference tasks affected by API execution failures and revised the associated results and discussion.

  2. Figure 4 (Page 7): We added a twelve-strategy heatmap and analysis across all twelve Buyers.

  3. Section 6.2 and Table 3 (Pages 8–9): We introduced a self-check baseline and included its experimental details in Appendix D.7: Single-Pass Buyer Self-Check Baseline (Page 24) and its prompt in Appendix F.8: Single-Pass Buyer Self-Check Prompt (Page 32).

  4. Appendix B: Candidate-Count Control (Page 19): We conducted Reference-10 experiments across eight Buyers to examine the effect of candidate-set size.

  5. Appendix B: Reference–Adversarial Outcome Transitions (Pages 19–20): We included bidirectional transition counts and case analyses examining decision changes and factors related to benchmark construction and evaluation.

  6. Appendix E: Training-Seed Robustness (Pages 25–26): We added checkpoint-selection curves and explained why the selected steps differ across training seeds.

Newly added text and updated numerical entries are marked in blue. We thank the reviewers and the Area Chair for their time and valuable feedback.

Dear Reviewers,

Thank you for your time and effort in reviewing this paper!

The authors have now posted their responses. I would like to kindly encourage you to review their rebuttal and engage in the discussion, particularly if any of your concerns remain unresolved.

Your feedback is greatly appreciated and will help us reach a fair and well-informed decision.

Best regards,
Area Chair, ICLR 2027

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.