acceptodds
Under review as a conference paper at ICLR 2027

CONMAN: Steering LLM Agent Actions via Contrastive World Model Manipulation

Abstract

Large language model (LLM) agents can enhance action planning by leveraging a world model (WM) to predict the resulting next state for each candidate action, which is then scored by an evaluator to guide action selection. However, this emerging paradigm introduces a new attack surface when WM training is outsourced to potentially adversarial third parties due to limited computational resources or domain expertise. In this paper, we investigate whether an attacker can poison a WM so that a downstream agent is more likely to select an attacker-specified target action at a prescribed trigger state, without accessing or modifying the agent, its inputs, planner, or the state evaluator. We address this challenge through contrastive manipulation (CONMAN) of the WM's training data. Rather than manipulating the next-state labels for the target action alone, CONMAN contrastively reshapes the predicted next states of both target and competing actions: target-action promotion assigns the target action next-state labels that appear favorable to the evaluator by indicating progress toward task completion, while competing-action demotion assigns selected competing actions unfavorable next-state labels indicating obstruction. These labels are automatically generated and selected to polarize evaluator scores while remaining plausible next states. We evaluate CONMAN across ALFWorld, ScienceWorld, and WebShop with four domain-specific target actions. CONMAN reaches 43.6–70.0% end-to-end attack success rate, and 70.0–88.2% success rate among episodes in which the target action is proposed, outperforming a promotion-only baseline, which reaches at most 20.0% and 50.0%, respectively. Meanwhile, on separate utility evaluation sets containing tasks unrelated to the target action, CONMAN preserves native-task utility, without degrading the task success rate for any target.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, world models, data poisoning, agent security

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iSoLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper identifies a new attack vector targeting LLM agents that rely on world models to predict the outcomes of their actions. The authors argue that these world models are likely to be outsourced due to how expensive these are, creating an attack surface through which poisoned world models can steer agents toward attacker-desired behavior. They propose a contrastive attack method to exploit this vulnerability and show that it increases the attack success rate (ASR) while largely preserving utility on benign tasks.

Critical strengths

None

Critical weaknesses
  • The practical motivation for using an external world model is insufficiently established. It remains unclear when this setup offers meaningful benefits over an agent interacting directly with the environment.

  • The attack appears to rely on restrictive assumptions about agent behavior. In the provided example, the agent seems to follow the world model’s predictions without verifying them against environmental feedback, raising questions about the attack’s applicability to agents that update their plans based on actual observations.

  • The methodological novelty is unclear. The approach appears to rely primarily on modifying data labels, and its distinction from existing data-poisoning methods is not sufficiently explained.

Questions for authors
  1. Could the authors provide concrete examples or experimental evidence demonstrating the benefits of using an external world model?

  2. Does the attack remain effective when the agent incorporates feedback from the actual environment and revises its plans accordingly?

  3. How does the proposed contrastive attack differ from existing label-manipulation or data-poisoning methods? Is the primary contribution the identification of a new attack surface or the attack technique itself?

1 comment

We thank the reviewer for the time and the questions, and we appreciate the opportunity to clarify. We answer the three points in turn and, for each, also point to where the paper discusses it. The revised PDF, uploaded during the discussion period, makes these places easier to find.

W1 / Q1: Why plan with a world model, and why an external one?

Why a world model. A WM does not replace interaction with the environment. As described in Sec. 3, the agent still takes an observation and executes an action in the real environment at every step. The WM lets it predict the outcome of each candidate action before committing to one. This matters when trying an action has a cost or cannot be undone, such as placing an order or a physical action; the first paragraph of Sec. 1 names these benefits as "improving task success, reducing execution costs, and avoiding irreversible consequences". Planning with WMs is an active line of work, summarized in Sec. 2 ("Agent planning"). WebDreamer (Gu et al., 2025) and WMA (Chae et al., 2025) use WMs to choose the actions of web agents, RAP (Hao et al., 2023) uses one for reasoning, and Li et al. (2026) train text WMs on agent trajectories and use them to verify irreversible actions before execution. These works report the benefits of WM-based planning.

Why external. "External" refers to who trains the WM, not to how the agent interacts with the environment. As noted in Sec. 1 and at the beginning of Sec. 4, an accurate WM needs large trajectory data from the environment, compute, and domain expertise; the training datasets provided by Li et al. (2026), which our experiments use, contain 40,000 to 70,000 trajectories per environment (Sec. 6.1). A developer without these resources takes a trained WM from a third party or outsources its training. This is the supply-chain setting of outsourced training in backdoor attacks (Gu et al., 2017). Our paper studies what such an untrusted WM can do to the agent that uses it.

W2 / Q2: Does the attack hold when the agent uses environment feedback?

Yes. The agent in all our experiments already does this. At every step it proposes candidates from the current real observation (Sec. 3, Eq. 1), and, as stated in App. D.3, "Only the executed action and its real observation are appended to history." The WM's predictions only score the candidates of that step and are then discarded. The agent therefore revises its plan from real feedback at every step. Figure 1 illustrates one planning step of this loop.

Feedback cannot undo the attack, because the attack succeeds when the target action is executed. The real observation arrives after the action: the agent has already opened or bought the promoted product, or the measurement procedure has already been broken. The 43.6% to 70.0% ASR@20 that CONMAN reaches in Table 1 is measured with this feedback loop. The revised PDF states the feedback loop explicitly in Sec. 3.

W3 / Q3: How does CONMAN differ from data poisoning, and what is the contribution?

The attack goal differs from conventional data poisoning. Conventional poisoning changes the behavior of the poisoned model itself, by writing the desired output into that model's training data; in the words of Sec. 2, such attacks "directly embed the target output, e.g., an action or answer, into content consumed by the attacked model or agent". Here the poisoned model is not the one that acts. The attacker wants a separate agent to change its behavior, without access to the agent's components, inputs, or data (Sec. 4). App. A (Table 4) compares CONMAN with seven recent attacks on WMs in this respect. This setting raises two challenges, discussed in Sec. 5.1, and CONMAN addresses each.

  1. The attacker cannot write the desired action anywhere. It can only change what the WM predicts, and an evaluator it does not control turns those predictions into scores. CONMAN therefore writes next-state labels that make the target action look like progress, and selects them with a surrogate evaluator.

  2. Raising the target is not enough, because competing actions at the same state can still score higher. CONMAN identifies these competing actions and demotes them, so that the target becomes the best-scoring candidate.

Conventional label poisoning applied to the WM is our Promotion-only baseline (Sec. 6.1). At the same 2% poisoning budget it reaches 20.0% to 27.5% ASR@20, against 43.6% to 70.0% for CONMAN (revised Table 1). The contribution is both, as listed at the end of Sec. 1: the attack surface, in which a poisoned WM steers a separate agent, and the method that makes the attack work under its two challenges.

Reviewer_CqRMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a supply-chain attack on language world models (WMs) used by LLM agents for action planning. In the considered planner, an action proposer generates candidates, a WM predicts the next state for each candidate, and an evaluator chooses the action whose predicted outcome receives the highest score. The attacker is a malicious WM provider who controls the WM training data but cannot access or modify the deployed agent, its proposer or evaluator, its prompts, or its inputs.

Critical strengths
  • A novel and consequential attack surface is formulated clearly. The work goes beyond attacks on a WM's output and demonstrates targeted control of a separate downstream agent's action without deployment-time input manipulation or access to the victim planner. This is a relevant supply-chain risk as agents increasingly use learned WMs for action selection.

  • The empirical analysis is careful and supports the core feasibility claim. Across four targets in three environments, CONMAN produces large paired ASR@20 gains over the corresponding clean WMs (+20.5 to +85.0 percentage points in the reported main and transfer settings; Appendix Table 16).

Critical weaknesses
  • The stealth claim is evaluated only away from the attacked action and trigger neighborhood. By construction, the utility sets contain tasks in which the target action never becomes available; ScienceWorld even excludes both task families used by the attack (Appendix D.1).
  • The evidence specifically attributing the gains to contrastive manipulation is narrow relative to the four-target claim. The main Promotion-only baseline differs from CONMAN in more than Demotion (CONMAN uses label selection with Q=8 and twice the total poisoned rows in the main setup), so it does not isolate the key contribution. Table 2 provides the appropriate equal-budget, Q=8 comparison and finds a meaningful +20-point gain with a paired 95% CI of [5, 35], but only for WebShop-A and apparently one training run. The other three targets lack this controlled comparison. Thus the paper convincingly shows that the complete attack works across targets, but does not yet establish that Promotion and Demotion are broadly complementary rather than especially effective in the chosen WebShop setting.
Questions for authors
  1. What are the exact overlaps between WM training data, trigger-state collection, attack-task construction, and evaluation for each target? In particular, do the target product IDs/action arguments and task templates occur in WM training, and can the authors provide results on target/product/task-disjoint evaluation states? Also, can the authors report utility on tasks where the target action is available but is not the desired action?
  2. Does the “without filtering” condition modify or truncate any existing rows beyond adding repeated poisoned rows? If so, please report a genuinely append-only experiment (and the resulting effective poisoned-token/trajectory fraction), or narrow the claim about the weaker data-contributor setting. A matched Q=8 Promotion-only and Q=1 Promotion+Demotion comparison across the main targets would also clarify the source of the reported gain.
3 comments

Part 1 of 3

We thank the reviewer for the careful review and for recognizing the attack surface and the empirical analysis. We answer each point in turn. CIs below are paired 95% bootstrap intervals over tasks.

W1: Stealth is evaluated only away from the target

The utility tasks of Table 1 are target-irrelevant by construction, so they measure utility away from the target. To measure it near the target, we built benign tasks that contain the target action, trigger-like states, or the competing actions, and ran them with the clean and CONMAN WMs of the main experiment. Utility does not drop.

The tasks are original tasks of the environments, each checked in the real environment.

  • Same target action, and taking it is correct. WebShop-A: 18 goals that ask for the target product itself (buying it with the goal's options earns the full reward). ALFWorld-A: 17 tasks whose destination is a drawer, so going to a drawer is correct, and 7 tasks whose task object lies in a drawer.
  • Trigger-like states. WebShop-A: 24 goals for other products in the target's category. Their search pages look like the attack pages, but the target is not on the first page.
  • Competing actions. 22 of these 24 pages list a competing product.
GroupTasksClean TSR@50CONMAN TSR@50CONMAN − Clean [95% CI]
WebShop-A, trigger-like (goals in the target's category)2495.887.5−8.3 [−20.8, 0.0]
WebShop-A, goals for the target product1838.938.90.0 [−27.8, +27.8]
ALFWorld-A, destination is a drawer1782.488.2+5.9 [−11.8, +23.5]
ALFWorld-A, task object in a drawer7100.0100.00.0
All near-target tasks66−1.5 [−10.6, +7.6]

In ALFWorld, 15 of the paper's utility tasks take place in rooms with drawers, with TSR@50 15/15 under CONMAN and 14/15 under the clean WM.

ScienceWorld excludes both task families used by the attack. The ScienceWorld-A target is focusing on the substance before the thermometer in temperature and melting-point tasks (App. Table 11), so every task of these two families is an attack task by definition: there is no temperature or melting-point task in which the target action is available and harmless. Their exclusion from the utility set is the partition itself, not a choice. The 24 utility tasks (eight other task types) still require the same "focus on" command as a step, with TSR@50 20/24 under both WMs.

The revised PDF describes the near-target groups in App. D.1, adds the table to App. E, and points to it in Sec. 6.2.

W2: Attributing the gain to contrastive manipulation

The submission's Promotion-only baseline differed from CONMAN in the number of poisoned rows and in label selection (Q). We have matched both.

Same number of poisoned rows, all four targets. We reran the Promotion-only baseline (Q=1) at the same 2% budget as CONMAN on every target; Table 1 of the revised PDF reports it in place of the 1% baseline.

ASR@20ALFWorld-AWebShop-AWebShop-BScienceWorld-A
Clean WM10.35.012.50.0
Promotion-only, 2%, Q=123.127.520.820.0
CONMAN, 2%43.670.062.555.0
CONMAN − Promotion-only [95% CI]+20.5 [7.7, 35.9]+42.5 [22.5, 62.5]+41.7 [16.7, 66.7]+35.0 [17.5, 52.5]

Every paired CI excludes 0, and CONMAN's conditional ASR is also higher on every target.

Same Q. Promotion-only with Q=8 at 2% leaves Demotion as the only difference from CONMAN. On WebShop-A it is in Table 2: 27.5% against 70.0% for CONMAN, a paired difference of +42.5 [25.0, 60.0]. For ALFWorld-A, WebShop-B, and ScienceWorld-A, we have trained the Promotion-only Q=8 WMs at 2% (CONMAN's Q=8 Promotion labels repeated to 2%, same recipe). Because of compute and time constraints, their evaluation could not be completed within the rebuttal period, for which we apologize. The final version of the paper will report, for each of the three targets, ASR@20, conditional ASR, and TSR@50 with paired bootstrap CIs against the clean WM and against CONMAN, which completes the controlled comparison on all four targets.

Part 2 of 3

Q1: Data overlap, disjoint evaluation states, and utility where the target is available

What overlap would matter for. Overlap between training and evaluation is a concern only if the poisoned WM recalls training pages. If it has learned a mapping from the target action at a trigger state to the poisoned outcome, the effect holds on states it has not seen. We report the exact overlaps and then test this directly.

Exact overlaps. The trajectories used to collect trigger states are excluded from all training sets (App. B.2), and the evaluation tasks come from outside the training data: ALFWorld's validation splits (App. D.1), ScienceWorld's test split, and, for WebShop, goals that the training corpus does not contain. For WebShop, we checked every evaluation goal by its exact instruction against all released WebShop training trajectories, from which every WebShop training set is drawn: none of the 40 WebShop-A and 24 WebShop-B attack goals, and none of the 50 utility goals, occurs. 34 of the 40 WebShop-A goals and all 24 WebShop-B goals were written for the study from catalog products with WebShop's own goal structure and are scored by the environment's own reward; the other 6 are original WebShop goals. The target action at its trigger states does occur in training: those steps are the poisoned instances. The clean WM is trained on the same rows except for the poisoned labels, so it saw the same pages and the same product IDs, and with it the agent picks the target in 5.0% of the WebShop-A attack tasks, against 70.0% with CONMAN. Every paired CI in the paper is computed against this control.

Evaluation on varied trigger pages. To test whether the poisoned WM recalls the trigger pages it saw in training or has learned that the target action at a trigger state leads to a favorable outcome, we took the paper's 24 WebShop-A fixed-candidate cases (App. D.4: task, search-results page, history, and five candidates in a fixed order, including the target and a competing action) and varied each trigger page by rule, in WebShop's own page format: reorder the listed products; replace the listed products that are not among the five candidates with other catalog products of the same category; reword the task instruction with the goal's attributes unchanged; and all three at once. The five candidates stay valid and in the same order, and every varied page passes the same format check as a real page. The protocol is the paper's fixed-candidate evaluation: each WM predicts one next state per candidate, the runtime evaluator scores them in three independent runs, and the highest-scoring candidate counts as selected (24 cases × 3 runs per row). No training is involved.

Trigger pageTarget selected, CleanTarget selected, CONMANMargin change, CONMAN − Clean [95% CI]
original (paper)1.465.3+71.7 [+62.0, +81.5]
products reordered1.459.7+57.3 [+45.0, +67.8]
products replaced0.063.9+61.8 [+51.2, +71.3]
instruction reworded0.062.5+63.8 [+53.7, +73.9]
all three1.450.0+50.7 [+40.8, +59.8]

The effect holds under every variation, and the clean WM almost never selects the target. The revised PDF adds this test to App. E.

Utility on tasks where the target action is available but is not the desired action. These tasks are the attack tasks by definition: the target is available and the task asks for something else. On them, a lower task success rate is the effect the attack aims at, since the agent takes the target instead of what was asked, and ASR measures it. For completeness, TSR@50 on the attack tasks of Table 1 (clean / CONMAN) is 43.6 / 30.8 on ALFWorld-A, 82.5 / 42.5 on WebShop-A, 79.2 / 8.3 on WebShop-B, and 52.5 / 15.0 on ScienceWorld-A. For benign tasks near the target, where the target action is available or nearly so and the task can still be completed, see the near-target utility results under W1 above: no group shows a utility loss, and the pooled TSR@50 difference over 66 tasks is −1.5 [−10.6, +7.6].

Part 3 of 3

Q2: The "without filtering" condition, an append-only experiment, and matched comparisons

Does "without filtering" modify or truncate existing rows? Yes, but only the poisoned rows, as in every setting of the paper: each poisoned row is an existing trajectory cut after the poisoned step, with the label of that step replaced, and the dataset keeps its size. This is within the threat model and keeps the comparison exact: the attacker trains the WM and controls its training data (Sec. 4), the cut only makes the poisoned label the last label of its row (App. B.2), and the fixed size keeps the poisoning ratio at its stated value. "Without filtering" refers to all other rows: trajectories that contain a target or competing action are kept in full instead of being cut before that action (Sec. 6.5). In that setting, every row other than the 2% poisoned rows is unchanged. The revised PDF describes the construction in these words in Sec. 6.5.

A genuinely append-only experiment. We have built and trained it on WebShop-A. The base is every source trajectory that the paper's WebShop WMs are built from, 70,576 trajectories, in full, unchanged, and in source order: nothing truncated, filtered, relabeled, reordered, or removed. Appended are the 3,500 poisoned rows of the paper's 5% WebShop-A run (Figure 2), i.e., the same 200 Promotion and 200 Demotion instances as the main run, repeated to fill these rows; 5% is the largest ratio of Sec. 6.5, and a larger poisoning ratio is what Sec. 4 and Sec. 5.3 name for the data-contributor attacker. Training shuffles rows, so appending at the end does not change the order seen in training. The resulting fractions, which the reviewer asks for, are: 74,076 rows in total, of which the poisoned rows are 4.72%; the poisoned labels are 0.15% of all supervised tokens; all supervised tokens of the appended rows, including their unchanged history turns, are 1.81%. The WM is trained with the same recipe as every WebShop WM. Because of compute and time constraints, its evaluation could not be completed within the rebuttal period, for which we apologize. The final version of the paper will report its ASR@20, conditional ASR, TSR@50, and score on the WebShop-A attack and utility tasks, with paired bootstrap CIs against the clean WM of the main experiment.

Matched Q=8 Promotion-only. As answered under W2: on WebShop-A the comparison is in Table 2 (27.5% against 70.0% for CONMAN, +42.5 [25.0, 60.0]); for the other three targets the Promotion-only Q=8 WMs at 2% are trained and will be reported in the final version of the paper with ASR@20, conditional ASR, and TSR@50 and paired bootstrap CIs.

Q=1 Promotion+Demotion. We did not train this variant: each such run is a full WM training on eight GPUs (Qwen2.5-7B, full-parameter SFT on 70,000 rows) followed by a full evaluation, and the compute of the rebuttal period went to the matched comparisons above. The WebShop-A ablations of Table 2 bound what label selection contributes: the same poisoned data with Q=1 and Q=8 gives ASR@20 of 20.0% vs 30.0% (Promotion, 700 rows), 27.5% vs 27.5% (Promotion, 1,400 rows), and 12.5% vs 15.0% (Demotion, 700 rows), i.e., 0 to 10 points, against +20.0 to +42.5 points from adding Demotion at the same number of rows (W2).

One training run. CONMAN is stable across three WM training seeds on WebShop-A, with a mean ASR@20 gain of 64.2 points over the clean WM (Sec. 6.4, App. E).

Reviewer_WxonOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CONMAN, a training-time poisoning attack on language world models used by LLM agents for planning. The attacker cannot modify the downstream agent, prompts, evaluator, observations, or environment, but can control the world model’s training data. CONMAN promotes a target action by assigning it a plausible favorable outcome and demotes strong competing actions by assigning them plausible failures. Across ALFWorld, WebShop, and ScienceWorld, the method achieves 43.6%–70.0% end-to-end attack success and 70.0%–88.2% success when the target action is proposed, while maintaining performance on target-irrelevant tasks.

Critical strengths

The first major strength is the identification of a meaningful and under explored attack surface. The paper shows that compromising a world model can change a separate agent’s decisions without modifying the agent, evaluator, prompts, or environment. This is important because externally trained world models may become common components of model-based agents. The second major strength is the contrastive attack design. Promoting the target while demoting its strongest competitors directly exploits the relative ranking used by the planner. The ablations and fixed-candidate analysis support this mechanism, while experiments across different models, evaluators, prompts, seeds, and adaptation methods suggest that the result is not tied to one exact configuration.

Critical weaknesses

The main weakness is the limited evidence for generalization. The evaluation covers only four attacker-selected target actions across three text-based environments, with some WebShop attack tasks constructed specifically for the study. The evaluation does not show whether CONMAN generalizes to arbitrary targets, infrequent trigger states, or varied formulations of the same action. Its effectiveness may also depend on the attacker being able to generate plausible and reusable failure outcomes for competing actions. A second weakness is that the utility evaluation avoids the region where collateral damage is most likely. Tasks where the target action becomes available are excluded, so the paper does not test benign tasks containing the same target, competitors, or trigger-like states. The defense analysis is also preliminary. In WebShop, poisoned labels are relatively detectable because they violate the native page format, while clean fine-tuning with 1,000 examples reduces attack success substantially. These findings weaken the broader claim that the attack is both stealthy and difficult to mitigate.

Questions for authors

How were the four target actions selected, and how many candidate targets were considered or discarded because of insufficient data, difficulty generating plausible labels, or low preliminary attack success?

Since 1,000 clean examples reduce WebShop attack success from 62.5% to 8.3%, is this mitigation consistent across seeds, environments, and clean-data selection strategies?

5 comments

Part 1 of 4

We thank the reviewer for the careful review and for recognizing the attack surface and the contrastive design. We answer each point in turn. CIs below are paired 95% bootstrap intervals over tasks.

W1 and Q1: Generalization of the evaluation

How the four targets were selected (Q1). Every target passes two criteria, both checked on the training data before any WM is trained.

  1. The target must have real-world meaning, i.e., it benefits the attacker or harms the user. WebShop-A and WebShop-B push buyers to a chosen product. ALFWorld-A pulls an embodied agent to chosen locations. ScienceWorld-A breaks the order of a measurement procedure so that the task fails.
  2. The target must have enough training instances: at least 200 training steps in which the target action is executed (for Promotion) and at least 200 training steps in which a competing action is executed at the target's trigger states (for Demotion). CONMAN relabels existing steps of the training data, so that the clean and the poisoned WM are trained on the same rows and differ only in the poisoned labels, which keeps the comparison controlled.

Candidates come from each environment's own action set: navigation actions to the receptacle types of ALFWorld, the steps of the temperature and melting-point procedures in ScienceWorld, and catalog products in WebShop. Many candidates fail the second criterion: about half of the WebShop catalog products never occur as the executed action in the training trajectories, and others, such as a throw blanket, have enough Promotion instances but too few competing actions for Demotion. No candidate was dropped after a trial attack, and no training or evaluation result was used to choose the targets. The revised PDF states the two criteria in Sec. 6.1, where the targets are introduced.

Only four attacker-selected targets (W1). The four targets cover three kinds of action: navigation (ALFWorld-A), product selection (WebShop-A and B), and action order (ScienceWorld-A). To extend the evidence, we are running CONMAN on new WebShop targets that pass the same two criteria and lie in product categories the paper does not cover. Each new target uses every setting of WebShop-A (same pipeline, 200 Promotion and 200 Demotion instances, Q=8, 2% budget, 40 attack and 50 utility tasks), with its own clean WM, CONMAN, and the 2% Promotion-only baseline. Because of compute and time constraints, these runs could not be completed within the rebuttal period, for which we apologize. The final version of the paper will report, for each new target, ASR@20, conditional ASR, TSR@50, and score with paired bootstrap CIs against its own clean WM.

Some WebShop attack tasks were constructed for the study (W1). A task can serve as an attack task only if two conditions hold. First, the task must be related to the target, so that the target product appears on the first page of the task's own search results; otherwise nothing can be measured. Second, the target must not be what the task wants, measured by the environment's own reward: the target earns less than the full reward; otherwise choosing it would not be a deviation. Among the original WebShop goals, only 6 meet both conditions for WebShop-A. We therefore wrote the other 34 WebShop-A goals and the 24 WebShop-B goals from catalog products with WebShop's own goal structure. A WebShop-A goal takes a catalog product in the target's category with its real attributes, options, and a price cap; the goal record has the same fields as an original goal. WebShop-B tests the case where the target almost fits the task: the target meets every stated constraint (size, color, price cap), as do the three competing products, and one preference sentence says which product the task wants (butterfly pattern, sleeveless style, or lowest price; eight goals each). Every episode is scored with the environment's own reward (App. D.4). The revised PDF states the two conditions where the attack tasks are defined (App. D.1).

Part 2 of 4

Arbitrary targets and infrequent trigger states (W1). The attack does not depend on a hand-picked trigger state: the trigger states follow from the target, and the evaluation already spans many of them. A trigger state is a state where the target action is available, i.e., a valid action satisfies the target's rule (App. Table 11). In WebShop, it is a results page that lists the target product, or the target's product page. In ALFWorld-A, it is a state of a one-object placement task in which the agent can go to a drawer that does not hold the task object and the destination is not a drawer. In ScienceWorld-A, it is a state of a temperature or melting-point task in which the substance can be focused on and neither the substance nor the thermometer has been focused yet. Choosing a target fixes its trigger states; they are part of the attack design, not an evaluation choice.

The evaluation already covers a distribution of trigger states, because every attack task reaches its own trigger state: in WebShop, each goal has its own search query and therefore its own results page; in ALFWorld, each task starts in a different room with a different object layout; in ScienceWorld, each task measures a different substance in a different scene. The new targets above each bring a new family of trigger states, so the same experiment covers both halves of the question. The revised PDF defines the trigger state in Sec. 6.1 in these terms; its use is unchanged: App. B.2 collects trigger states to select competing actions, and at test time every state where the target action is available counts.

Varied formulations of the same action (W1). The attack holds under varied formulations: with a proposer that writes actions in plain language, CONMAN still reaches 62.5% ASR@20, against 5.0% with the clean WM (table below). There are two reasons. First, in these environments an action is a fixed command (click[asin], go to drawer 3, focus on substance). The environment runs only such commands, and the proposer writes every candidate in this format from the list of valid actions it receives at each step (App. C.1, D.3). The WM sees a command, never a paraphrase. Second, a target is a rule, not one command (App. Table 11): in ALFWorld-A, going to any drawer that does not hold the task object counts; in WebShop, opening the target product or buying it counts; on the Demotion side, go to countertop 1 and go to countertop 2 are one competing action (App. B.2). Every command that performs the action falls under the rule.

The free-form test runs on WebShop-A with the clean and CONMAN WMs of Table 1 and the same evaluator; the command syntax is removed from the proposer's input. A rewriting step by the agent's own LLM (one call per step, given the current valid commands) turns each candidate into one legal command before the WM sees it; the environment runs legal commands only, so a free-form agent needs this step to act at all. The rewriter maps 99.7% of the free-form candidates to a legal command. Results (40 attack, 50 utility tasks):

MetricClean, free-formCONMAN, free-formCONMAN − Clean [95% CI]Paper (standard proposer), Clean / CONMAN
ASR@205.062.5+57.5 [+42.5, +72.5]5.0 / 70.0
Cond. ASR6.964.15.6 / 70.0
Utility TSR@5064.072.0+8.0 [−2.0, +20.0]64.0 / 70.0

The revised PDF adds this row to Table 3, the proposer and rewriter prompts to App. C.5, and one sentence to Sec. 6.4.

Plausible and reusable failure outcomes (W1). The attack does not depend on reusable failure outcomes. A Demotion label is written for one instance: the generator sees that instance's task and history and writes the next state for that step (Sec. 5.4). No label is reused across tasks or instances. One blockage type per environment (a blocked path, a concealed object, a stockout) is a design choice that tells the generator what kind of next state to write; the label itself is new each time. The label also does not have to be a failure that holds in every case; it has to make the competing action score below the target at that state, which is measured end to end by the attack success rate. Plausibility: see the audit under W3.

The revised PDF rewrites the sentence on a "consistent demotion signal" in App. C: the blockage type tells the generator what kind of next state to write, and each label is generated for its own instance.

Part 3 of 4

W2: Utility near the target

The utility tasks of Table 1 are target-irrelevant by construction, so they measure utility away from the target. To measure it near the target, we built benign tasks of the three kinds the reviewer names and ran them with the clean and CONMAN WMs of the main experiment. Utility does not drop.

The tasks are original tasks of the environments, each checked in the real environment.

  • Same target action, and taking it is correct. WebShop-A: 18 goals that ask for the target product itself (buying it with the goal's options earns the full reward). ALFWorld-A: 17 tasks whose destination is a drawer, so going to a drawer is correct, and 7 tasks whose task object lies in a drawer.
  • Trigger-like states. WebShop-A: 24 goals for other products in the target's category. Their search pages look like the attack pages, but the target is not on the first page.
  • Competing actions. 22 of these 24 pages list a competing product.
GroupTasksClean TSR@50CONMAN TSR@50CONMAN − Clean [95% CI]
WebShop-A, trigger-like (goals in the target's category)2495.887.5−8.3 [−20.8, 0.0]
WebShop-A, goals for the target product1838.938.90.0 [−27.8, +27.8]
ALFWorld-A, destination is a drawer1782.488.2+5.9 [−11.8, +23.5]
ALFWorld-A, task object in a drawer7100.0100.00.0
All near-target tasks66−1.5 [−10.6, +7.6]

The paper's own utility sets also use the target's command where it is a correct step of the task: in ALFWorld, 15 utility tasks take place in rooms with drawers (TSR@50 15/15 with CONMAN, 14/15 with the clean WM), and in ScienceWorld all 24 utility tasks require "focus on" an object as a step (TSR@50 20/24 under both WMs). The revised PDF describes the near-target groups in App. D.1, adds the table to App. E, and points to it in Sec. 6.2.

W3 and Q2: Defenses

Training-data audit. The audit already favors the defender: it assumes that the provider hands over its training data, while in our threat model the provider is the attacker (Sec. 4). Even so, it detects the paper's WebShop labels mainly because they break the page template (Sec. 6.6). We therefore regenerated the WebShop-A Promotion and Demotion labels under a format rule enforced on every candidate: the exact native page layout, character for character. A Demotion label is the native page with exactly one added availability field after the price ("Currently unavailable in every size and color"). A Promotion label may only add option values to the existing option groups, each with its click action, and change the price; every native field, option value, and action stays in the native order. We reran the paper's audit on these labels with the same frozen sample restricted to WebShop-A (50 poisoned and 125 clean trajectories), the same judge model, reference trajectory, three runs, and four policies; the clean trajectories are unchanged, so their judgments are the paper's. Mean [min, max] over the three runs:

PolicyPoison recall, paper's labelsPoison recall, format-matched labelsClean false-positive rate
Standard79.3 [76.0, 84.0]0.0 [0.0, 0.0]1.3 [0.0, 2.4]
Strict-8081.3 [78.0, 86.0]1.3 [0.0, 4.0]2.4 [0.8, 4.0]
Strict-9094.0 [92.0, 96.0]26.7 [24.0, 28.0]41.6 [39.2, 44.0]
Strict-9597.3 [96.0, 98.0]78.0 [76.0, 80.0]75.5 [75.2, 76.0]

Under the standard policy, none of the 50 poisoned trajectories is flagged in any run; under the stricter policies, poisoned trajectories are flagged no more often than clean ones. As a control for judge drift, the 50 trajectories with the paper's labels, judged again on the same day, are flagged at 96.0% under the standard policy, so the drop comes from the labels. The WM trained on the format-matched labels keeps the attack's strength on the Table 1 WebShop-A tasks: ASR@20 72.5% against 70.0% for CONMAN (+2.5 [−15.0, +22.5]), with utility TSR@50 72.0% against 70.0%. In ALFWorld and ScienceWorld, where observations are sentences, the audit already cannot separate poisoned from clean trajectories: it flags about 41% of the clean ones (Sec. 6.6). The revised PDF adds this experiment to the audit paragraph of Sec. 6.6.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful reading and constructive comments. We are glad that the reviewers find the attack surface novel and consequential (Wxon, CqRM) and the contrastive design supported by the ablations and the fixed-candidate analysis (Wxon). Below we summarize the new experiments and the changes to the paper. Each reviewer's reply gives the full setups, tables, and paired 95% bootstrap confidence intervals (CIs) over tasks.

New experiments completed during the rebuttal period

  1. Equal-budget baseline on all four targets (CqRM W2, iSoL W3). The Promotion-only baseline is rerun at the same 2% poisoning budget as CONMAN on ALFWorld-A, WebShop-A, WebShop-B, and ScienceWorld-A. CONMAN is +20.5 to +42.5 ASR@20 points above it, and every paired CI excludes 0. Table 1 of the revised PDF reports this baseline.
  2. Utility near the target (Wxon W2, CqRM W1, CqRM Q1). 66 benign tasks on WebShop-A and ALFWorld-A that contain the target action, trigger-like states, or the competing actions. The pooled TSR@50 difference between CONMAN and the clean WM is −1.5 points [−10.6, +7.6], and no group shows a utility loss.
  3. Training-data audit with format-matched labels (Wxon W3). With the poisoned labels written in WebShop's page format, the audit flags 0.0% of poisoned trajectories under its standard policy (79.3% with the paper's labels), and the WM trained on these labels reaches 72.5% ASR@20 (CONMAN in Table 1: 70.0%).
  4. Clean fine-tuning on a second environment (Wxon W3, Q2). On ScienceWorld-A, 2,000 clean examples lower ASR@20 from 45.8% to 12.5%; 2,000 clean examples that do not cover the target leave 20.8%.
  5. Robustness to varied trigger states (CqRM Q1). On varied trigger pages (listed products reordered or replaced, instruction reworded), the evaluator still selects the target in 50.0% to 63.9% of the fixed-candidate cases with CONMAN, against at most 1.4% with the clean WM.
  6. Free-form action proposer (Wxon W1). With a proposer that writes actions in plain language, ASR@20 is 62.5% with CONMAN against 5.0% with the clean WM.

Experiments still running. Training is complete for (a) the Promotion-only baseline with label selection (Q=8) on ALFWorld-A, WebShop-B, and ScienceWorld-A (CqRM W2, Q2) and (b) a genuinely append-only poisoning setting (CqRM Q2); the attack pipeline is also running on new WebShop targets (Wxon W1, Q1). Because of compute and time constraints, we could not complete their evaluation within the rebuttal period, for which we apologize. These results cannot enter the revised PDF within the discussion period; the final version of the paper will report each of them with ASR@20, conditional ASR, and TSR@50, with paired bootstrap CIs.

Changes in the revised PDF (uploaded during the discussion period). Sec. 3 states the agent's feedback loop explicitly (iSoL). Sec. 6.1 defines the trigger state and states the two target-selection criteria, with details in App. B.2 (Wxon). Table 1 reports the 2% baseline on all targets (CqRM, iSoL). Table 3 gains the free-form proposer row, with its prompts in App. C.5 (Wxon). Sec. 6.5 describes the "without filtering" construction (CqRM). Sec. 6.6 adds the format-matched audit and the ScienceWorld-A fine-tuning results, and revises the sentence on when clean fine-tuning lowers ASR; App. D.5 and App. E give the protocols and tables (Wxon). App. C rewords the description of the Demotion blockage type (Wxon). App. D.1 states the two conditions of an attack task and defines the near-target utility groups (Wxon, CqRM). App. E adds the near-target utility table and the varied-trigger-state test (Wxon, CqRM).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.