- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
he paper proposes Human Before the Loop, an agent architecture that replaces open-ended tool calls with a developer-authored hierarchical extended finite-state machine (HEFSM). The LLM selects among semantically meaningful menu options, while fixed Python routines execute the corresponding actions; the design aims to make sensitive data and dangerous operations unreachable by construction. The authors evaluate the approach on prompt-injection attacks, benign task completion, and the Pokémon Red tutorial. They report zero successful exfiltrations across 328 attacks, an increase in average task completion from 76.5% to 81.4%, and successful tutorial completion by a 30B model through the constrained interface where its tool-calling run made no progress.
Critical strengths
1-Moving the security boundary from runtime detection to developer-defined permissions and executable actions is a compelling approach: if sensitive resources truly cannot be reached through any action path, the system need not rely on the model correctly identifying every novel injection. This is the paper’s most consequential contribution.
2-The paper does more than report attack-blocking results: it also measures benign task completion and tests a capability-heavy task. The Pokémon comparison, in particular, illustrates how semantic action abstractions can reduce low-level interaction overhead and change what a model can accomplish through a harness.
Critical weaknesses
1-The proposed system cannot access arbitrary files or perform destructive actions or payments, while the OpenClaw baseline has a broader action surface. That restriction makes the reported zero exfiltrations plausible by construction, but it also means the comparison does not establish that the framework secures an otherwise equivalent agent. The benign evaluations likewise use a limited set of supported actions, including a filtered subset of GAIA prompts. The paper would be stronger if it quantified the functionality excluded by the design and evaluated security and utility at matched task coverage.
2-The attack evaluation uses 328 fixed attacks and four models; the reported successful exfiltrations occur only with one model, and the study does not establish robustness to adaptively optimized attacks. The reliability and capability results also rely on relatively small, specialized task sets, with no uncertainty estimates reported in the presented results. This leaves unclear how stable the performance gains are and how well they generalize beyond these settings.
Questions for authors
1-What precisely is the security guarantee’s scope? In particular, does “unreachable” cover only locally stored canary data that the agent is prohibited from reading, or also sensitive information the agent might encounter through permitted web or API actions? How are information flows through the predefined scripts and outbound channels accounted for?
2-How much task functionality is excluded by the constrained design relative to OpenClaw? Can the authors quantify the proportion of evaluated requests that require capabilities the constrained agent does not provide, and clarify how task coverage was matched across the two systems?
3-Were the reliability results repeated across independent runs or seeds, and can the authors report uncertainty estimates? Given the small task sets, this would help determine whether the reported gains—especially the average increase—are robust rather than sensitive to particular prompts or runs.
Thank you for taking the time to review Human Before the Loop!
We agree that our security evaluation did not clearly show the difference our framework was making, so we added a run with a least-privilege OpenClaw that was also separated from the local data. While its exfiltration rate was 0%, its benign task completion fell to 66.2%, while ours stayed at 100% (Sec. 5.2, Fig. 2).
As for your second concern, while many models have been post-trained not to fall for prompt injections, the dynamic nature of evolving attacks does not guarantee against exfiltration, and our framework does provide this guarantee: "Model-level resistance is not guaranteed, as attack success depends heavily on the corpus and whether the attack was adaptively optimized (Ma et al., 2026; Hofer et al., 2026). Bounding the action space in the framework makes the model choice irrelevant to exfiltration." (Sec. 5.3)
We have included a new threat model section that covers exactly what our contribution protects against (Sec. 3). We focus only on local data exfiltration, as we feel this is a threat many users are not expecting, and out of the box many general-purpose agents have access to local data for convenience even when it is not needed.
In the reliability evaluations, we matched the task capability between the two frameworks. In our original submission, it was never clearly stated that OpenClaw was simplified down and our framework was scaled up to match the same actions the synthetic and GAIA sets were testing. This is now clearer (Sec. 6.1, App. E.2, App. E.3).
Due to the breadth of our evaluations across three sections, we did not rerun the evaluations. However, we feel this is important, so we have added easy-to-run, fully reproducible harnesses for every eval in our paper, linked below the conclusion in our Reproducibility Statement.
We computed 95% confidence intervals from the existing runs: the improvement over OpenClaw is +5.7 points [+1.5, +9.8] on the synthetic set and +4.9 points [+0.5, +9.0] across both sets, so the gain holds beyond the specific prompts, while the GAIA subset alone (31 prompts) is too small to separate the two frameworks.
Other updates: simplified title, clearer figures (Fig. 1), a flattened-menu run in the reliability eval (Sec. 6.2, App. D.5), a new Pokémon graphic in the appendix (App. D.6, Fig. 5), full menu breakdowns for each eval (App. E), and an updated repo with new data and the harness for the flattened menu.