- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates branch steering in computer-using agents, where an attacker modifies untrusted runtime data to force the agent into a pre-approved branch that favors the attacker. The authors propose COBRA, a system that generates fixed programs and branch constraints via a trusted planning model, utilizes an isolated vision model for perception, and enforces constraint checks through a Branch Resolution Hub and HTTP/MCP proxies. The paper's contributions include STEER-Bench, a constraint enforcement architecture spanning GUI and MCP interfaces, MESA (for generating candidate site descriptions), and a planning method that integrates multiple candidate plans. The authors report a 0% Attack Success Rate (ASR) across 199 attack configurations in STEER-Bench and demonstrate the prevention of in-scope attacks across four external security benchmarks; they also evaluate utility and cost using 361 OSWorld-MCP tasks. While the research direction is valuable, the current evidence most directly supports the prevention of observable requests that violate pre-generated policies; significant gaps remain regarding claims of robust protection against branch manipulation.
Critical strengths
-
The choice of execution boundaries is well-founded and offers practical system value. Section 4 implements checks at the level of HTTP requests and MCP calls, enabling the system to inspect targets, endpoints, and parameters rather than inferring operational semantics from click locations. The shared constraint state also directly addresses privilege bypass issues that arise when GUI interactions are intermixed with tool invocations.
-
The component ablation and protection scope analyses are highly informative. STEER-Bench conducts separate ablation studies on components such as branching constraints, permission refreshing, and tool definition integrity, while also providing benign reference cases. The comparison of WASP performance with and without sitemaps clearly illustrates the differences in capability between domain-level and endpoint-level restrictions. These results provide valuable insight into the system's dependencies. The experiments were conducted thoroughly.
Critical weaknesses
W1. The definition of DFI is stronger than the guarantees actually described by the system; the veracity of branch conditions is not resolved by type checking.
First, the paper defines DFI such that untrusted runtime data cannot influence which authorized branch is executed, nor determine security-critical parameters. However, Section 4 explicitly allows attacker-supplied values to select a branch, provided constraints are met. Yet, Appendix A describes the following process:
- Q-VLM returns a value from an untrusted observation;
- Format and type are checked;
- The value is bound to a program variable;
- The branch condition is computed using deterministic code;
- Constraints for that branch are activated.
Thus, under the described process, the branch still satisfies:
Computing deterministically does not guarantee the veracity of . A value that is well-formed and within a valid numerical range—but derived from display content forged by an attacker—could still trigger the wrong branch.
This point is particularly critical because the paper's own definition of an attack is: an attacker does not introduce new actions but instead induces the system to enter an already-approved, dangerous branch. Guaranteeing that execution does not exceed a branch's privileges once entered does not automatically prove that the basis for entering that branch is trustworthy.
W2. The output of the trusted planning model is treated as the security policy, yet the correctness of the policy itself and the correctness of its execution are not verified separately.
Appendix A of the paper explicitly states that deterministic parsing can only extract program structure; it cannot determine exactly which operations the user permits at each step. This implies that specific permissions are generated via P-LLM annotations. Consequently, the system relies on at least two distinct conditions:
- The agent accurately executes the generated policy;
- The generated policy accurately reflects user authorization.
The paper describes the first condition thoroughly but lacks separate verification for the second.
For instance, if the P-LLM omits a resource constraint, permits an overly broad set of objects, or erroneously approves an alternative branch, the deterministic agent could execute this flawed policy with perfect correctness. In this scenario, a "generation constraint violation" does not occur, yet the user's intent may still be contravened.
This issue also touches upon evaluation criteria; the paper defines a successful attack as one that violates currently active constraints. If the success label relies on constraints generated by the system itself, "harmful behaviors that align with an erroneous policy" might be overlooked.
W3. Ambiguity regarding security implications of metadata changes: Halt execution or downgrade to domain-level restrictions?
The main text claims that if an approved sitemap or tool definition changes, a hash mismatch halts execution, preventing the dispatch of untrusted actions.
However, Appendix E describes a different scenario: missing, invalid, or unapproved sitemaps place the site under host-level enforcement; similarly, if an approved sitemap changes and awaits re-approval, the site remains under domain-level protection.
The following behaviors carry distinct security implications:
- Immediately halting the current task;
- Continuing execution while retaining the old version's endpoint constraints;
- Discarding endpoint constraints and reverting to a domain-level whitelist.
The paper fails to clearly distinguish the stages and conditions under which these behaviors apply.
This is not a trivial matter of phrasing: the paper's own WASP experiments demonstrate that with only domain-level protection, 14 out of 19 same-site unauthorized action attacks succeeded. If a metadata change triggers the third behavior, the level of protection could degrade significantly.
W4. Strong security outcomes and broad utility outcomes rely on different protection conditions; no conclusions have been established regarding the security-utility trade-off under a unified configuration.
WASP results indicate that preventing same-site unauthorized actions requires an approved sitemap. Conversely, the broad utility experiments in OSWorld-MCP explicitly utilize a setup without sitemaps, where HTTP protection is limited to the host level.
Consequently, task completion rates across a broad range of tasks do not directly answer the question: How much utility is retained when the paper's critical endpoint and field constraints are enabled?
Additionally:
- In Table 2, the performance gains observed for UI-TARS stem from the integration of an additional GPT-5 planning model and cannot be attributed solely to the security architecture itself. - When the same model assumes both roles, the performance of kimi-k2.5 drops from 42.1% to 25.5%, and Sonnet-5 drops from 51.0% to 38.8%.
- Appendix A allows for follow-up queries to the user regarding safety-critical values during planning, yet it fails to report the frequency of such human interventions or whether the baseline was granted the same capability.
- The 69.2%/72% figures in MESA represent the coverage of required actions; they do not account for metrics such as field description errors, overly broad permissions, or auditing costs.
W5. The evaluation thoroughly validates component-level checks against established constraints but falls short of verifying robustness against adaptive branching manipulation.
STEER-Bench constructs attacks on a component-by-component basis and measures the Attack Success Rate (ASR) after disabling the corresponding component checks. While this design effectively answers whether a specific check functions in these scenarios, it fails to adequately address:
- Can an attacker discover an erroneous path that simultaneously satisfies existing constraints?
- Can the attacker exploit combined effects across steps or channels?
- Do automatically generated policies remain correct when encountering attacks for which they were not specifically designed?
The threat model explicitly prohibits attackers from inspecting, modifying, or adapting to submitted plans. While the inability to modify a protected plan is a reasonable boundary, the concept of "inability to adapt to the plan" requires further definition: Are updates to the attack based on observable execution feedback permitted? Are multiple attempts allowed? The paper lacks a sufficiently clear description of the budget and process for adaptive attacks.
External validation offers a valuable supplement; however, the proxy check experiment for OS-Harm—which explicitly instructs the ReAct agent to follow injected content—primarily tests the agent's ability to block specific requests rather than the robustness of the overall system's policy during a natural attack sequence.
Furthermore, although the paper discusses ceLLMate and other permission-based constraint methods, it fails to provide a comparison with key alternative systems using identical approval metadata, models, and budgets. Consequently, the current results do not clearly distinguish whether the benefits stem from endpoint access control, branch state management, or COBRA's holistic planning approach. Impact: Affects the role of existing ablation components; the broad claim of "eliminating branch steering" still requires more targeted attacks and comparative evidence.
W6. Core statistics contain verifiable inconsistencies, affecting the interpretability of baseline comparisons
Evidence: Table 1 and lines 291–320; Contributions statement (p. 2); External validation (p. 7).
| Item | Verification Result | Implication |
|---|---|---|
| ASR(off)=87.6% | Based on the number of cells per suite and the rounded percentages, the reconstructed success counts are 26, 24, 10, 72, 18, 4, 6, 10, and 4, totaling 174/199=87.4%. | The 87.6% figure can be obtained by rounding percentages for each row before weighting, but this cannot replace the aggregate of raw success counts. While the discrepancy is minor, the calculation method requires correction or clarification. |
| General baseline: 74 attack cells, yet 94.4% and 89.5% reported | Based on the binary outcome definition ("one cell = one configuration run"), there is no integer such that rounds to either of these values. | This may involve unspecified repetitions, weighting, or different denominators. Table 1 does not list the actual number of evaluations per baseline row; the key comparisons in the summary cannot currently be reconstructed. |
| 1,259 attacks in the out-of-scope range | 1,259 equals exactly 21+19+1,210+9; however, the main text states that all 51 OS-Harm attacks were blocked by the full system. If all were included in the full system's coverage scope, the total would be 1,289. |
Questions for authors
-
What exactly does DFI guarantee? If two different untrusted observations both pass type checking but cause the program to select different branches, does the system still satisfy your definition of DFI? If so, please provide a definition consistent with lines 169–174; if not, what specific deterministic mechanism prevents this scenario?
-
How does S5 PROV verify the source rather than just the value? Please present a complete case study: when a trusted source and attacker-controlled content provide the same, validly formatted value, what verifiable evidence does the system rely on to distinguish between them? Who generates the source information, and can the Q-VLM forge or misidentify it?
-
Is the determination of a successful attack independent of the strategy generated by the P-LLM? If the generation strategy omits user constraints and an attack happens to align with that strategy, is it counted as a successful attack? Please explain the criteria for this determination, as well as how strategy errors, manual corrections, and failed plans factor into the final statistics.
-
What is the precise runtime behavior after an approved sitemap changes? Does the system terminate, retain old constraints, or switch to domain-level protection? Do the main text and Appendix E describe different stages? Please provide results covering these state transitions, particularly the behavior when endpoint protection can no longer be maintained.
-
How does the shared constraint state handle cross-channel duplicate effects and delayed requests? For example, if an MCP call actually completes an operation but returns an error, leading to a subsequent GUI fallback, the same operation might be executed again. Does the system bind permissions to the number of operations, specific program steps, or unique requests? Do S4, S8, and S9 cover scenarios where "individual calls are valid, but the combined effect exceeds authorization"?
-
Can you report on security and utility under the same strong protection configuration? Please report—for the same tasks—the ASR, benign completion rate, and completion rate under attack, as well as the frequency of manual approvals, supplemental key value inputs, and sitemap revisions under endpoint/field constraints; does the baseline receive the same trusted information?
11 comments
Response to Reviewer (2/4)
Weakness 3
Thank you for pointing out the need to distinguish these transitions. We apologize for the ambiguity between blocking changed metadata and terminating the entire task. A missing, invalid, or initially unapproved sitemap provides no endpoint-level authority, so the website is handled under host-level restrictions. When retrieved metadata differs from a previously approved version, the hash mismatch prevents the changed version from being accepted under the old approval and triggers a new approval request.
The outcome then depends on that decision. If the new sitemap is approved, it can support endpoint-level constraints. If it is not approved, the website remains accessible under host-level protection. For MCP, an unapproved tool definition makes the tool unavailable.
We agree that host-level fallback provides weaker protection, as the WASP results demonstrate. We have clarified what the hash check blocks in the Threat Model (§3) and described the approval and fallback behavior in Appendix E (Sitemap-Conditioned HTTP Guarantees).
Weakness 4
Thank you for raising this point. The broad OSWorld-MCP results measure utility with domain-level HTTP restrictions. STEER-Bench and WASP also measure benign completion with stronger protections enabled on a narrower set of tasks. In WASP, approved sitemaps allowed COBRA to block 14 additional same-site attacks while preserving benign completion on the corresponding tasks.
We evaluated utility with endpoint and field checks enabled on the website-facing Chrome subset of OSWorld-MCP. As reported in Appendix D, MESA-generated sitemaps cover all required operations for 18 of the 25 website-facing tasks. Both configurations used GPT-5 as the planner, UI-TARS-1.5-7B as the vision model, and pass@5. In the original evaluation with host-level enforcement, COBRA completed 11 of those 18 tasks. With the MESA-generated sitemaps and endpoint and field checks enabled, COBRA still completed 10 of those 11 tasks (90.9%). We have added this result to §5.2.
The model comparison in Table 2 points to a distinction between planning and grounding. A larger general-purpose model can be better suited to planning the task in advance, while a smaller model trained for GUI grounding can be better suited to locating and interacting with elements on screen. This is the rationale for pairing GPT-5 with UI-TARS, consistent with the observations in CaMeL-NOVA. The improvement over UI-TARS alone reflects the performance of that pairing and cannot be attributed to security enforcement alone.
Response to Reviewer (3/4)
Weakness 5
Thank you for raising these questions. Regarding robustness against adaptive attacks, in our experiments, we concentrate on validating individual checks. Constructing adaptive attacks in this setting is non-trivial. The constraints are set by user intent, so the bounds are intended to be similar to what a human user would have set for themselves when navigating these websites. Adaptive attacks would therefore need to walk a narrow line between a benign execution and a malicious execution that still satisfies these constraints. This requires predicting the user’s agent’s plan and its constraints well enough to exploit a gap. The attacker cannot directly inspect the committed plan, although they could try to infer it from observable execution feedback. Even if they succeed, it is unclear whether enough room remains within the constraints to cause actual harm rather than merely reach the bounds of what the user authorized. Evaluating this would require a substantial additional attack effort beyond the predefined attacks considered here.
Regarding whether an attacker can discover an erroneous path that satisfies existing constraints, COBRA checks the resulting operation against the constraints for the active path. As discussed in W1, steering the agent toward another outcome the user has authorized can change what happens while still satisfying the task’s requirements. If an undesirable outcome nevertheless satisfies the enforced constraints, this could reflect either a specification problem, where the user did not sufficiently constrain the task, or a policy generation problem, where the generated policy fails to capture a requirement the user specified. We discuss the correctness of automatically generated policies in W2.
Regarding the meaning of “inability to adapt to the plan,” we have clarified this wording in the threat model. The committed plan and interpreter state are protected from direct inspection and modification. This does not prohibit attackers from observing external behavior, inferring which branches and limits apply, or updating attacks based on execution feedback. However, the attacker may have only one opportunity to get the attack right for a particular plan: different users and tasks can produce different plans and constraints, so feedback from one user’s plan does not necessarily generalize to another.
Regarding the scope of the OS-Harm proxy experiment, we agree that it addresses a narrower question: whether the proxy blocks unauthorized HTTP requests when the agent attempts them. It does not evaluate the robustness of the overall system during a natural adaptive attack sequence.
Regarding what COBRA adds beyond endpoint access control, ceLLMate constrains which operations are permitted but does not provide control-flow integrity (CFI). As discussed in §6 (“Why constrain both control flow and data flow?”), consider the request “Buy the Brand A product if it is available, spending at most €50. If it is unavailable, buy an alternative for at most €40.” Keeping the permissions for both branches active throughout the task would allow a €45 checkout on the fallback branch, even though the user authorized only €40 for that outcome. COBRA associates constraints with the committed plan and activates the €40 limit on the fallback branch, rejecting that checkout. Enforcing this distinction requires linking permissions to the active execution path, beyond endpoint access control alone. S8 isolates this contribution by comparing branch-specific updates with keeping the union of permissions active throughout the task: ASR is 0% with updates and 100% with fixed permissions in that suite.
Regarding combined effects across steps or channels, S9 tests how constraints combine across nested branches, and S4 tests whether the same active constraints are enforced across HTTP and MCP. Together with S8, these ablations provide targeted evidence for branch state management and enforcement across channels, although they do not exhaustively evaluate adaptive attacks exploiting combined effects across steps or channels.
Response to Reviewer (1/4)
We thank the reviewer for the careful assessment and for recognizing the practical value of HTTP/MCP enforcement and shared constraint state. We also appreciate the positive evaluation of the ablations, benign reference cases, and sitemap comparison.
Weakness 1
Thank you for raising this distinction. A well-formed observation can be false and can lead the agent to select a different branch. The Branch Resolution Hub determines the constraints for the selected path, and the HTTP and MCP proxies check the outgoing operation against them. The scope of this protection depends on which conditions the user specifies and how faithfully the plan captures them.
For example, suppose the user asks the agent to buy a product from an online shop for a price between USD 30 and USD 40. An attacker without access to the user’s prompt or the committed plan would have to infer or guess the upper limit to target it. Suppose the attacker guesses correctly and controls the shop, so they can set the price to USD 40. COBRA checks the actual charge in the checkout request and allows it. In this case, the attacker has influenced the price, but this is not a successful attack, as “buy the product at USD 40” is still an outcome the user explicitly authorized. If instead the user specifies exactly USD 30, COBRA rejects the same USD 40 checkout request because it violates the price constraint.
We have clarified this distinction in the Threat Model (§3) and Architecture (§4). COBRA enforces the bounds expressed in the plan, including any conditions required to authorize the operation. Whether those constraints faithfully capture the user’s authorization is the separate question we address in W2.
Weakness 2
COBRA generates its constraints before execution using the trusted user request, the plan, and approved site and tool descriptions. For example, if the user specifies a €100 spending limit, the prompt supplies the limit and the sitemap identifies the checkout field to which it applies. If the planner determines that the trusted inputs are insufficient to set a critical constraint, it asks the user for clarification before committing the plan. We agree that a set of constraints can be enforced correctly and still omit a requirement from the user’s request. The correctness of the generated constraints still depends on how faithfully the planner translates those inputs into constraints. These are similar limits to correctness of automatic constraint generation that the Dual-LLM methods that we derive these methodologies from (CaMeL, Fides) have as well, if we do not assume a human to do the constraint generation.
We evaluated constraint generation on 30 STEER-Bench cells (§5). We manually annotated the expected constraints before generating them with COBRA, then used probes to check whether the generated constraints admitted authorized operations and blocked unauthorized ones. A cell counted as correct if every probe yielded the expected decision. The generated constraints were correct in 28 of 30 cells. In the two remaining cells, the constraints were too restrictive in one and too permissive in the other.
Regarding the evaluation criteria, STEER-Bench defines the harmful outcome for each task and checks whether it occurred using the actions recorded by the environment. For example, if an attack aims to send a payment to an unauthorized recipient and that payment goes through, it counts as a successful attack even if the generated constraints omitted the recipient restriction. We have clarified this scoring criterion in the STEER-Bench design paragraph (§5).