- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLM-driven agents can bypass behavior-based CAPTCHAs such as Google reCAPTCHA v3. The authors propose Seqion, a multi-agent framework built on browser-use that plans task-oriented actions and injects human-like behaviors, including curved mouse trajectories, keystroke delays, and non-uniform scrolling. Experiments on a self-hosted testbed and an external website show that Seqion obtains higher trust scores than the RL-based baseline of Abstract. Its scores exceed those of human users on the testbed and approach them on the external website. The authors conclude that fine-grained behavioral synthesis by agentic AI can defeat behavior-based CAPTCHAs.
Critical strengths
-
The topic is timely and practically important. Understanding how robust frictionless CAPTCHAs are against capable LLM agents matters to the security community.
-
The evaluation includes an external real-world website with human comparison, not only a self-hosted testbed.
Critical weaknesses
-
The central "granular" claim is not supported. In Table 5, Bézier trajectories score lower than straight lines on all four backbones. The main gain comes from having mouse movement at all, which is a coarse signal.
-
The behavioral effects are mostly small, inconsistent, and possibly confounded. Gains vary in direction across backbones, and the largest occur on abnormally low baselines. Static factors rival or exceed most behavioral effects: changing the IP raises the mean score from 0.52 to 0.83 (+0.31), and adding cookies raises it from 0.19 to 0.51 (+0.32); in the single-behavior ablations (Tables 5–7), every gain larger than this starts from a baseline of 0.30 or below. Repetition penalties, IP history, and session duration are not controlled, and no significance tests are reported.
-
The baselines do not isolate the claims. Abstract [1] is task-agnostic, has few reproduction details, and its results shift sharply between environments (0.69 vs 0.34). There is no comparison with trajectory-level methods (e.g., Akrout et al. [2]; BeCAPTCHA-Mouse [3]), and no non-LLM scripted baseline to show what the agentic components contribute.
[1] Tsingenopoulos et al. Captcha Me If You Can: Imitation Games with Reinforcement Learning. EuroS&P 2022.
Questions for authors
Q1. Are the mouse and keyboard events trusted (e.g., via CDP) or synthetic dispatchEvent events? Can the authors confirm that reCAPTCHA actually observes the simulated behaviors?
Q2. Can the authors rerun the behavioral ablations in a controlled environment: fixed IP pool, clean browser profile, randomized condition order, and a duration-matched control? Please report pass rates with significance tests.
Q3. How does Seqion compare with a non-LLM scripted baseline using the same behavior library, and with trajectory-level methods [2, 3]?
[2] Akrout et al. Hacking Google reCAPTCHA v3 Using Reinforcement Learning.
[3] Acien et al. BeCAPTCHA-Mouse: Synthetic Mouse Trajectories and Improved Bot Detection.
2 comments
Thanks for the response. The rebuttal solves some of my concerns. My other concerns remain:
- rrweb capture does not show that v3 scores these events. Appendix L also generates no keydown/keyup events at all.
- No reruns were done, and the order of conditions within each module is undisclosed, with no randomization reported. Nominally comparable no-behavior settings give 0.52, 0.24 and 0.19 (Tables 5, 3 and 8), and there is no duration-matched control. The significance test covers Seqion vs. Abstract, not the ablations.
Therefore, I am inclined to keep my score.
W1 & Q1: Both mouse trajectories and keystrokes are executed via Tool Agent–generated JavaScript injected into the browser (e.g., dispatchEvent(new MouseEvent(...)) and input events with randomized delays). We confirm reCAPTCHA v3 observes these events because rrweb (which hooks standard DOM listeners like v3) captures the exact trajectories and intervals (Appendix K), and ablating these injected JS events immediately drops scores back to the 0.14–0.52 baseline (Table 5). We agree that Table 5 shows the primary gain comes from trajectory presence rather than curve complexity (with heuristic Bézier curves introducing wider curvature variance than human paths in Fig. 9d). Crucially, this reveals that reCAPTCHA v3 relies heavily on coarse spatiotemporal event presence and pacing rather than strictly enforcing isTrusted===true or complex trajectory kinematics.
W2 & Q2: First, single-behavior variations across tasks (Tables 5–7) reflect task-modality alignment (e.g., scroll helps on Blog Reading up to +0.39 rather than non-scrollable Signup), and gains are not limited to low baselines (on GPT-5-mini's 0.52 baseline, mouse movement reaches 0.70, and full Seqion reaches 0.78—a +0.54 gain over R1's 0.24 under a fixed IP without cookies, exceeding single static factors). Second, our experiments enforced strict environmental and temporal controls (to be documented in §4): except for the cookie ablation, browser profiles were fully reset before each run under a fixed IP and site key, with a 48-hour cooldown interval between experimental modules to prevent cross-condition repetition penalties, and identical on-page interaction time intervals across variants (the 111s vs. 251s in Table 3 reflects off-page LLM planning latency). Third, because trust scores are discrete ordinal values ({0.1, 0.3, 0.7, 0.9}) and consecutive trials across the 100-run sequence experience cumulative repetition effects, we applied the non-parametric Wilcoxon signed-rank test paired by trial index: Seqion achieves a 90% pass rate (score >= 0.7) on the testbed and 73% (vs. 21% for Abstract) on the external site, with Seqion's mean score (0.60) significantly exceeding Abstract (0.34) on the external site (Wilcoxon signed-rank p = 3.2e-12, rank-biserial r = 0.94).
W3 & Q3: While a non-LLM scripted baseline using the same behavior library performs similarly to R3 (0.71 mean score, Table 3) on known static selectors, it fails to generalize across arbitrary DOMs without manual selector rewriting and lacks task-aware topological ordering (R4 improves R3 from 0.71 to 0.78 under identical interaction pacing). This also explains why task-agnostic Abstract drops from 0.69 (testbed) to 0.34 (external e-commerce site). Furthermore, prior trajectory-level methods [2, 3] optimize isolated 2D pointer paths (e.g., for v2 checkboxes) without semantic task planning, typing rhythm, or scroll coordination; as Tables 5–7 show, v3 continuous verification requires matching the primary interaction modality of the task (mouse, keystroke, and scroll) rather than mouse curvature alone. We will add [2, 3] and this discussion to §2 and §4.