acceptodds
Under review as a conference paper at ICLR 2027

Human Before the Loop: Constraining the Action Space Improves Security, Reliability, and Capability in AI Agents

22 views

Abstract

Multipurpose AI agents are increasingly common, and so are the data breaches they cause. These agents are vulnerable to an attack called prompt injection, which can instruct a large language model (LLM) to find and deliver sensitive data to an attacker. Current proposed defenses carry their own trade-offs. Post-training a model to refuse malicious instructions fails to predict novel or unseen attack vectors. Live supervision slows down runtime operation. Security middlewares can only stop attacks they are built to recognize. Defenses that restructure an agent’s control flow have been documented to have drops in benign task completion. To address this, we present Human Before the Loop, an agent framework that determines the action space and data access in the design phase, leaving no runtime path open to sensitive data or destructive actions. Rather than relying on traditional open-ended tool calling, our framework uses a method we term semantic action abstraction, which navigates actions within a Hierarchical Extended Finite State Machine (HEFSM). This eliminates the chance of data exfiltration while increasing benign task success and realized capability. In our evaluations, our framework prevented all prompt injection attacks, whereas the baseline tool-calling agent failed to prevent 8.5% of attacks and delivered sensitive data to the attacker. Benign task completion rose from 76.5% with the tool-calling agent to 81.4% with our framework. Additionally, our framework enabled a 30B parameter model to complete the Pokémon Red tutorial phase in 341 turns, while the same model using a tool-calling harness made no progress beyond the initial stage across 1,000 model calls. That 30B model finished the tutorial in 22% fewer model calls than the larger model it was distilled from. This method supports the creation of autonomous agents that can be trusted to run fully unsupervised, without a trade-off between security and capability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI agent security, data exfiltration, AI Safety, autonomous agents, trustworthy AI, LLM agents, finite state machines, prompt injection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bEQ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

he paper proposes Human Before the Loop, an agent architecture that replaces open-ended tool calls with a developer-authored hierarchical extended finite-state machine (HEFSM). The LLM selects among semantically meaningful menu options, while fixed Python routines execute the corresponding actions; the design aims to make sensitive data and dangerous operations unreachable by construction. The authors evaluate the approach on prompt-injection attacks, benign task completion, and the Pokémon Red tutorial. They report zero successful exfiltrations across 328 attacks, an increase in average task completion from 76.5% to 81.4%, and successful tutorial completion by a 30B model through the constrained interface where its tool-calling run made no progress.

Critical strengths

1-Moving the security boundary from runtime detection to developer-defined permissions and executable actions is a compelling approach: if sensitive resources truly cannot be reached through any action path, the system need not rely on the model correctly identifying every novel injection. This is the paper’s most consequential contribution.

2-The paper does more than report attack-blocking results: it also measures benign task completion and tests a capability-heavy task. The Pokémon comparison, in particular, illustrates how semantic action abstractions can reduce low-level interaction overhead and change what a model can accomplish through a harness.

Critical weaknesses

1-The proposed system cannot access arbitrary files or perform destructive actions or payments, while the OpenClaw baseline has a broader action surface. That restriction makes the reported zero exfiltrations plausible by construction, but it also means the comparison does not establish that the framework secures an otherwise equivalent agent. The benign evaluations likewise use a limited set of supported actions, including a filtered subset of GAIA prompts. The paper would be stronger if it quantified the functionality excluded by the design and evaluated security and utility at matched task coverage.

2-The attack evaluation uses 328 fixed attacks and four models; the reported successful exfiltrations occur only with one model, and the study does not establish robustness to adaptively optimized attacks. The reliability and capability results also rely on relatively small, specialized task sets, with no uncertainty estimates reported in the presented results. This leaves unclear how stable the performance gains are and how well they generalize beyond these settings.

Questions for authors

1-What precisely is the security guarantee’s scope? In particular, does “unreachable” cover only locally stored canary data that the agent is prohibited from reading, or also sensitive information the agent might encounter through permitted web or API actions? How are information flows through the predefined scripts and outbound channels accounted for?

2-How much task functionality is excluded by the constrained design relative to OpenClaw? Can the authors quantify the proportion of evaluated requests that require capabilities the constrained agent does not provide, and clarify how task coverage was matched across the two systems?

3-Were the reliability results repeated across independent runs or seeds, and can the authors report uncertainty estimates? Given the small task sets, this would help determine whether the reported gains—especially the average increase—are robust rather than sensitive to particular prompts or runs.

1 comment

Thank you for taking the time to review Human Before the Loop!

We agree that our security evaluation did not clearly show the difference our framework was making, so we added a run with a least-privilege OpenClaw that was also separated from the local data. While its exfiltration rate was 0%, its benign task completion fell to 66.2%, while ours stayed at 100% (Sec. 5.2, Fig. 2).

As for your second concern, while many models have been post-trained not to fall for prompt injections, the dynamic nature of evolving attacks does not guarantee against exfiltration, and our framework does provide this guarantee: "Model-level resistance is not guaranteed, as attack success depends heavily on the corpus and whether the attack was adaptively optimized (Ma et al., 2026; Hofer et al., 2026). Bounding the action space in the framework makes the model choice irrelevant to exfiltration." (Sec. 5.3)

  1. We have included a new threat model section that covers exactly what our contribution protects against (Sec. 3). We focus only on local data exfiltration, as we feel this is a threat many users are not expecting, and out of the box many general-purpose agents have access to local data for convenience even when it is not needed.

  2. In the reliability evaluations, we matched the task capability between the two frameworks. In our original submission, it was never clearly stated that OpenClaw was simplified down and our framework was scaled up to match the same actions the synthetic and GAIA sets were testing. This is now clearer (Sec. 6.1, App. E.2, App. E.3).

  3. Due to the breadth of our evaluations across three sections, we did not rerun the evaluations. However, we feel this is important, so we have added easy-to-run, fully reproducible harnesses for every eval in our paper, linked below the conclusion in our Reproducibility Statement.

We computed 95% confidence intervals from the existing runs: the improvement over OpenClaw is +5.7 points [+1.5, +9.8] on the synthetic set and +4.9 points [+0.5, +9.0] across both sets, so the gain holds beyond the specific prompts, while the GAIA subset alone (31 prompts) is too small to separate the two frameworks.

Other updates: simplified title, clearer figures (Fig. 1), a flattened-menu run in the reliability eval (Sec. 6.2, App. D.5), a new Pokémon graphic in the appendix (App. D.6, Fig. 5), full menu breakdowns for each eval (App. E), and an updated repo with new data and the harness for the flattened menu.

Reviewer_tXRoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Human Before the Loop, a defense against prompt injection for AI agents. The proposed technique works by pruning the action space before the agent runs; because an expert developer oversees this process, the authors claim that their technique is secure against prompt injection attacks. The authors evaluate benchmarks like AgentDojo and find that their scheme prevents all attacks. At the same time, Human Before the Loop can maintain the same utility as the undefended agent across a variety of tasks, including one long-horizon task involving a video game.

Critical strengths
  1. I think that system-level defenses against prompt injection are an important research direction, and I appreciate the authors’ contribution to this line of work. Most system-level defenses come with some guarantees on security at the cost of utility. Thus, this scheme’s ability to preserve (and in some cases improve) task utility is quite notable.
  2. The authors evaluate their scheme across a variety of different models and model providers (i.e., OpenAI and Meta). This demonstrates that the proposed scheme is not tied to any particular model (i.e., it is model agnostic). I also like that the authors focus on recently released models; the work thus operates in a very contemporary setting.
Critical weaknesses
  1. There is no section in the manuscript that formalizes the threat model. I understand at a high level that the authors want to consider how prompt injection can be used for data exfiltration, but I think there should be a dedicated section on the threat model (perhaps in Section 2 or 3) to properly ground the security claims in the paper.

    • Without a formal threat model, it is difficult to evaluate the scheme’s proposed security in context. For instance, the authors mention in Section 3.1 that web search is allowed. Could this be a potential attack vector? For instance, could a maliciously placed Google Form online make its way into agent context, and eventually cause the agent to exfiltrate sensitive data?
  2. Along the same lines, there is no section that explicitly outlines the security properties of the proposed scheme. Do the authors claim that Human Before the Loop is secure-by design due to human-driven pruning of the action space? If so, this should be stated clearly in a dedicated section. I don’t think there is any need for math and/or proofs, but I think there should be something more concrete than just written prose; a formal claim or an algorithm’s pseudocode could be helpful here.

  3. I think the current discussion of related work is incomplete and should be expanded. In particular, the authors reference CaMeL, but don’t discuss any other work on system-level defenses against prompt injection:

    At the very least, I think that the authors should have a brief discussion on some of these and explain how their proposed method is different and/or addresses the limitations present in previous schemes. The authors should also better situate their contributions within the context of existing work; for instance, the multiple-choice selection design in Section 3.1 is similar in style to the usage of enums for prompt injection security in [3] and the usage of booleans in [1]. I think this would better motivate the work and help make the core contributions of the submission clearer.

  4. There is no discussion on how the defense is implemented for the tasks evaluated in this paper. I think it would be useful to have a separate section (before Section 4) that sketches out the human-designed decision trees for a couple tasks, along with a written walkthrough. The section could incorporate specific design decisions and explain in-depth how the implementation process differs from task to task.

Questions for authors

Thank you for submitting your work to ICLR 2027. I think this is interesting work, and I like the focus on building a system-level defense. The framing reminds me a bit of how interactive voice response (IVR) systems operate; for instance, when a user calls their bank, they select options on a numeric keypad in order to get routed to the correct department. This work seems to apply a similar principle to prevent agents from being misled by prompt injection. However, in its current state, I think there are several points that should be addressed to improve the quality of the submission. First, the threat model and security claims should be better specified, and the relevant assumptions should be better explained. Next, the current discussion on related work is insufficient and should include more recent work on system-level defenses. Finally, the authors should have an in-depth section that explains how they apply their scheme to the tasks evaluated in this paper. In addition to this, I have the following questions about the work:

  • How involved is the effort for the human expert? If the human involvement is substantial, this will significantly impact the deployability of the proposed scheme in practice. At the moment there are only a couple sentences on this in the limitations section; I encourage the authors to be transparent about how involved human pruning is, and to discuss any tricks that can expedite the process within the authors’ chosen tasks.
  • I think the Pokemon Red evaluation is interesting, but I am surprised that the baseline performs so poorly. Is there any particular reason why Human Before the Loop performs much better compared to the baseline harness? Was the objective provided to the agent to complete the tutorial as fast as possible? If not, I imagine that there could be other factors at play (i.e., the stronger Muse Spark 1.2 model might have taken the initiative to perform orthogonal but benign tasks, such as leveling up the starter on Route 1). I think a more detailed discussion on what exactly led to this surprising result would be helpful.
  • It looks like the submission doesn’t perform a standard evaluation of AgentDojo. Many existing prompt injection security papers look at the entirety of AgentDojo, and report performance on each of the individual environments (i.e., Workplace, Slack, etc.). Is there any particular reason that the authors chose not to do so for this submission?
4 comments

Thank you for taking the time to review Human Before the Loop! With the additional page allowance for the rebuttal phase, we were able to add some of your suggestions, and we feel it really strengthened the paper.

For your first concern, we added a dedicated threat model section above the methodology (Sec. 3). Originally there was a bit in the introduction, but it never specified that we would be solving local data exfiltration, so we made that clearer.

For your second point, we believe this section in the introduction explicitly states the contribution (Sec. 1). Rather than pruning out sensitive data, the developer determines all fixed actions and never adds sensitive data access in the first place:

"This paper's contribution is an agent architecture in which the set of reachable actions is fixed before the agent runs, rather than applying a further defense during runtime. A developer designs the state machine, actions, and data the agent can reach, placing all human oversight in the design phase and leaving the agent free to run without supervision."

Additionally, we added full menu/system breakdowns for each of the evaluations in the appendix (App. E), along with fully reproducible harnesses in a repo linked below the conclusion (Reproducibility Statement), which addresses your concern in #4.

We really appreciate the additional resources you shared with us. We were fighting with the page limit originally, but we now had room to integrate system-level defenses into our related work section under prompt injection defenses (Sec. 2, App. A.1).

  1. Since the human expert builds every action in the action space, it scales with the amount of effort put in: with less time, a developer could create a smaller, more specialized agent, and this scales up with more effort. We also feel the trade-off is fair between a developer creating an agent and the users who get to use it safely without human-in-the-loop supervision. We expanded our discussion section to mention this (Sec. 8).

  2. We intentionally used a capability eval that has historically been shown to have weak baselines. The harness we used was actually intended to be used with online self-improvement, where the instructions and prompts evolve as the model attempts to progress, and even models with visual capabilities struggle with visual reasoning. The Claude Plays Pokémon experiment took Claude over a year to complete the game (App. A.5).

Our evaluation is run without any self-improvement, so the baseline tool-calling harness struggles (Sec. 7.1).

Many agent capability benchmarks measure generalization (the number of tasks an agent can complete), and we really wanted to showcase how a base model's capability could be improved by supporting the model's strengths rather than fine-tuning out its weaknesses.

  1. As for AgentDojo, throughout the paper we used simplified evals rather than full benchmarks to showcase every angle of the framework, prioritizing breadth over a single full benchmark (Sec. 1, Sec. 5.1). For security, we felt that showing more 0% exfiltrations in a case where exfiltration is impossible would be redundant.

We also wanted to bring up some other updates we made to the paper as a result of the other reviews.

Security eval: we ran a least-privilege OpenClaw to separate the difference our framework was making, and saw benign task completion fall to 66.2%, while ours stayed at 100% (Sec. 5.2, Fig. 2).

Capability eval: we gave the tool-calling harness our semantic actions to see how much of the work the menu was doing, and while there was an improvement, it wasn't as efficient as the full framework (Sec. 7.2, Fig. 4).

Other updates: simplified title, clearer figures, a flattened-menu run in the reliability eval (Sec. 6.2, App. D.5), a new Pokémon graphic in the appendix (App. D.6, Fig. 5), and an updated repo with new data and the harness for the flattened menu.

The clearer methodology figure (Fig. 1) also shows the full system better, with an example of how the system works in practice; previously it was difficult to make out.

Thank you for your response. I think the paper's presentation has certainly improved. Unfortunately some of my concerns from earlier remain. More specifically:

  • Threat model and core contribution. The new Section 3 and the paragraphs at the end of Section 4.1 are helpful in establishing the scope of the work. However, they do not directly address and/or consider threats to the security of the proposed defense (for instance, the web search example I discussed earlier). The claim in this work is that Human Before The Loop is secure-by-design, and it makes intuitive sense why this is the case. But such a claim, in my opinion, needs a more in-depth technical discussion than what is currently present in the manuscript.

  • Extent of human involvement. I appreciate the additions made to the appendix. But given that expert human design is a core part of this work, there needs to be a more detailed discussion in the manuscript. It is still not clear how much effort is expected out of the expert human designer. I encourage the authors to think about different metrics that could be used to convey this (i.e., rough estimate for number of hours needed, number of states in the associated HEFSM, number of fixed actions in the defended agent vs. the undefended agent, etc.). Not all of these might be relevant, but I think something quantitative is needed here.

Thus, I will be unable to increase my score.

Thanks for the follow-up, and we’re glad the presentation reads better now!

As for the web search concern: web content can still end up in the agent’s context, so an injection can happen, but all it can do is nudge which menu option gets picked. None of the options can touch sensitive data or send arbitrary content anywhere (like submitting to a malicious Google Form). A search query can only contain what’s already in context, and by design there’s nothing sensitive there. The same goes for any untrusted input, not just the web, which is why we handled it at the framework level instead of channel by channel.

On how much work the human side takes: the framework has also been used to build a larger, fully featured system comparable in scope to OpenClaw, built by a single developer over about four months alongside this paper and its evaluations. That system is publicly released, with no security incidents to date. We had concerns that discussing the released system and its development timeline in the paper would compromise anonymity, so we kept both out. We also felt the main text was better spent showing what this kind of framework makes possible for capability and reliability, rather than on a full code breakdown of an implementation that’s already publicly available. We’re happy to include those details in the camera-ready, should the paper make it that far.

Thanks for the comment; I appreciate the authors' responsiveness. While the OpenClaw-style system seems intriguing, unfortunately I can only review what is currently present in the submission. I continue to encourage the authors to think about methods to quantitatively measure human effort for the case studies presented in the paper (i.e., AgentDojo, Pokemon Red, etc).

Reviewer_7DVrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The work proposes an agent framework in which a developer authors a HEFSM at design time. At runtime, the model may only pick from enumerated menu options via constrained JSON schema output, with each pick bound to pre-written deterministic code. Because the model never emits a tool call, a command, or free-form arguments like paths and URLs, no path through the machine reaches sensitive local data. Results are provided on prompt-injection attacks across four models vs. OpenClaw, benign task completion on a synthetic prompt set plus a GAIA Level 1 subset, and a Pokemon task as a realized-capability test.

Critical strengths
  1. The proposed framework is able to eliminate the risks, with limited or even positive benign task performance differences on evaluated tasks.
Critical weaknesses
  1. The contribution is not shown to require the architecture. No comparison against restricted tool calling (only safe tools) or a flat allowlist enum (all actions allowed all the time). As the security guarantee comes from the closed action set alone, the HEFSM and "semantic action abstraction" framing are asserted rather than ablated. OpenClaw and similar harnesses already support granular tool permissions and handle tool-call parsing, so "restricted tool access" is a standard alternative.

  2. Pokemon result does not separate harness engineering from the framework. The constrained agent reads the map directly from the emulator, receives a menu of nearby interactable objects, and gets auto-walk, obstacle avoidance and dialogue advancement in Python written by people who knew the map, while the baseline gets screenshots, absolute coordinates and button presses and was deliberately run with its online adaptation disabled. This measures how much of the task was absorbed into code rather than which model reasons better.

  3. The security result is close to tautological. Zero leaks is what the design specifies, so the experiment confirms the implementation rather than measuring the defense's effectiveness.

Questions for authors
  1. Is there any experimental evidence that replacing JSON prior-turn outputs with plain text is beneficial? [L227]

  2. Does re-serializing prior assistant turns as text break KV-cache reuse, and if so what is the measured cost?

  3. What does HEFSM contribute over a single flat menu of actions; is there any ablation?

  4. Is web browsing able to navigate arbitrary external sites? If so, could sensitive data be exfiltrated by entering it into a page or URL?

2 comments

Thank you for taking the time to review Human Before the Loop! We felt many of your points were valid, and we filled the gaps in our evaluation that didn't clearly show where the framework was making the difference.

In the security eval, we added a least-privilege OpenClaw that is also separated from the sensitive data. While it had the same 0% exfiltration rate, its benign task completion fell to 66.2%, while ours stayed at 100% (Sec. 5.2, Fig. 2).

While zero leaks are expected, prior work often sees a drop in benign task completion, and this is where our framework doesn't make a compromise.

For our capability eval (Pokémon), we added a run where the tool-calling harness also had the same semantic actions as our framework, to separate the HEFSM from our semantic actions (Sec. 7.1). It performed better than tool calling alone, but Glimmer still failed to finish, stalling in Oak's Lab until the 1,000-call cap, and Spark took 178 calls at 0.78 USD compared to 58 calls at 0.15 USD with our framework (Sec. 7.2, Fig. 4). The eval itself was not meant to be a showcase of model capability, but of the realized capability an agent can have when combined with programmed actions. No instructions for how the game should be played were added, only nearby interactable objects, and the prompt was to finish the tutorial (Sec. 7.1, App. E.5).

Question 1. During the engineering phase, we noticed that smaller models would nest the format when JSON was in previous turns. We found that stripping it down to plain text fixed this. This is noted in our methodology (Sec. 4.2, App. B).

Question 2. There are no issues with the KV cache. Each previous turn is converted to plain text once, when it is added to the history, and never changed after, so each prompt is an exact prefix of the next one and caching still works. We checked our logged runs, and the full history carried over unchanged in 1,016 of 1,016 follow-up calls in the security eval and 330 of 330 in the synthetic set. The prompts are also small: 585 input tokens per call on average in the security eval and 298 in the synthetic set.

Question 3. We used a hierarchy for design practicality, as a flattened menu would grow the size of the input prompt as it scales. However, your question got us curious, so we ran an additional flattened-menu run in the reliability eval. The results were very similar, slightly better on GAIA and slightly worse on our synthetic dataset, and this has been added with a more detailed result report in the appendix (Sec. 6.2, App. D.5, App. E.4).

We also added our rationale to the methodology section of the paper (Sec. 4).

Question 4. Our agent can browse the web but has no state that accesses sensitive data, and our paper focuses on local data exfiltration rather than web data exfiltration. We added a new threat model section to detail exactly what we seek to protect against (Sec. 3, Sec. 5.1).

Other updates: simplified title, clearer figures (Fig. 1), a new Pokémon graphic in the appendix (App. D.6, Fig. 5), full menu breakdowns for each eval (App. E), and an updated repo with new data and the harness for the flattened menu.

Thank you for your response and revisions.

The flat versus nested results are interesting - I agree that it would likely matter more as number of options scale.

The risk of web browsing leaking sensitive information relies on the agent not having sensitive information in context. But this can conflict with the desired tasks: agent can send a private message (has private message info in context), and it could also leak that private info via a web search. There is a tradeoff between security and utility, and though the proposed method helps security, it doesn't necessarily guarantee leak prevention.

The presentation and experiments are improved, but I still have concerns around the tradeoff of human time required to design HEFSM versus generic restricted agent settings.

For these reasons I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.