- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that CAPTCHA solving should be judged by whether the answer comes from a task-consistent interaction process beyond the final answer. To test this, the authors build HLL in a local Android/Chrome simulator: 10 task families and 33 evaluation cells (400 instances each), varied along three axes: webpage distraction, harder variants, and dynamic validation. Dynamic validation requires the recorded trace to pass rules on drags, clicks, repeated errors, and state transitions. The key result is a ranking reversal: Claude-Opus-4.6 leads the static setting (90.0%), while Gemini-3.1-Pro leads under dynamic validation (45.0%, vs. 26.3% for GPT-5.4 and 23.8% for Claude-Opus-4.6). From this, the authors conclude that answer-only evaluation can overstate agents’ verification ability. The appendix stress-tests the validators through human calibration, legality audits, and trace repair, with supplementary runs under AppAgent and UI-TARS. The authors position HLL as a simulator-level diagnostic.
Critical strengths
-
Separating answer correctness from interaction acceptance is a useful diagnostic lens for GUI agents, and the controlled design makes this diagnosis credible: across ten task families, all models within a cell are evaluated on the same 400 seeded instances. The failure analysis in the main text is informative: weaker models mainly fail at perception and target identification, while stronger models more often reach the correct answer but fail at grounding, ordered interaction, or stateful execution. The authors also give a preliminary, single-model example of how the diagnosis can guide improvement: on 50 paired tile-restoration instances, explicit state recording solves 26 versus 19 for generic planning.
-
Rather than assuming the validators are reliable, Section 4.2 tests whether the validators are reliable. Human participants pass in 239 of 240 trials, and every trace with removed or corrupted required records is rejected. The paper also reports the validators' failures: legal small-step and center-click strategies are wrongly rejected in 1 of 20 and 3 of 20 cases, and 1 of 10 illegal state transitions is missed. Section 4.5 adds an equal-budget comparison of answer-only and full validation, plus a trace-repair study, so readers can judge which conclusions depend on specific validator settings.
-
The paper also checks the robustness of its rankings. Section 4.5 reports that Gemini-3.1-Pro keeps its lead when any single family is removed, while the order of GPT-5.4 and Claude-Opus-4.6 can flip. Gemini-3.1-Pro also stays highest when families are grouped by validation mechanism. The authors further rerun three models under AppAgent, add a UI-TARS run on the same instances, and state explicitly that these rankings describe complete agent configurations rather than isolated harness effects.
Critical weaknesses
-
The dynamic layer is itself under-validated, so dynamic scores have no clear meaning. The paper's central finding, that static strength does not reliably transfer to dynamic acceptance, rests on these rules, yet their support is thin. The only human behavioral reference is three team members, and task legality is judged without an explicit criterion. The paper's own evidence is also in tension. Scripted small-step drags are treated as a legal strategy and rejected only once in 20 trials, yet 16 of the 43 rejected-but-correct episodes sampled for the repair study arise from agents reaching the correct position through several small drags, each below the displacement threshold. The paper does not say whether these 16 are legal. Combined with trace edits reversing 30 of 31 sampled geometric rejections once thresholds are known, this suggests the geometric rules check the form of the record rather than the legality of the interaction. How much of the dynamic score reflects genuine violations versus validator artifacts is therefore unknown. So is the meaning of results such as GPT-5.4 solving 18 slider instances under answer-only checking but only 1 under full validation at equal budgets.
-
Each realism axis changes several things at once, so performance shifts cannot be attributed to the factor it names. The axes are meant to locate where, and under which conditions, the pipeline fails, which requires changing one variable at a time. Yet distraction changes layout and decoys together, and results on the same task swing both ways: on slider alignment, GPT-5.4 rises from 68.50% to 100%, while GLM-5V falls from 75.50% to 0%. The hard axis changes task content and budget together. Static and dynamic results come from separate rollouts under different budgets, with an equal-budget control for only one model on two families. The authors acknowledge in Section 4.4 that these conditions are composite, but then the design cannot answer the question it was built to answer.
-
The main figures are of poor quality. Figures 1–3 are blurry and their small text is hard to read, which makes the benchmark overview difficult to follow. They give the impression of having been produced with generative tools without careful review. The authors should replace them with clear, publication-quality versions.
Questions for authors
-
Two pieces of evidence don't line up. Section 4.2 treats scripted small-step drags as legal, rejecting them once in 20 trials. Yet 16 of the 43 sampled rejected-but-correct episodes come from agents reaching the target in several small drags, each below the displacement threshold. If such drags are legal, these are false rejections, and the 1-in-20 figure may understate real ones. If not, why does reaching the right position in steps violate the task? What exactly defines task legality?
-
Claude-Opus-4.6 scores 96.75% on static tile restoration but 0/400 dynamically. The audit finds illegal interactions in 9 of 10 audited episodes, 8 of them "trajectory-related," yet the tile-restoration rules in Appendix A.4 include no trajectory check. Which rule rejects these submissions, and does Opus use a different input mode, such as drag instead of tap? If the harness's drag wrapper is the cause, this 0 reflects the environment, not the model.
-
The only budget-controlled comparison of answer-only and full validation covers GPT-5.4 on two families: 18 versus 1 on slider and 10 versus 3 on tile restoration. AppAgent results, meanwhile, are reported only for the dynamic setting. Could you extend the comparison to a few representative models and families, and add static results under AppAgent? If the ranking reversal survives, the central claim becomes much stronger.
8 comments
Part 2
Regarding the 16 small-step dragging cases in the repair study, we have reexamined these cases and confirmed that they reached the correct final position, but the recorded sequence of short drag gestures did not satisfy the predefined displacement threshold. The rejections were therefore triggered by this geometric condition. Accordingly, these cases should be explicitly described as "answer-correct but non-compliant with the geometric requirements of the full dynamic protocol." The relevant rules are documented in Appendix A.4 (p. 19) of both the original and revised manuscripts, while the repair mechanisms are described in Appendix B.6.7 (p. 35) of the original manuscript and Appendix B.6.7 (p. 37) of the revised manuscript.
Table 20 (p. 35) of the original manuscript and Table 26 (p. 38) of the revised manuscript report that, while preserving the submitted answer and final position, trace editing recovered 22/31 geometric rejections using default parameters and 30/31 using predetermined threshold-aware parameters. Here, "recovery" means that the modified record passes the predefined dynamic checks. It does not imply that the original record satisfied those checks.
We acknowledge that these results reveal a limitation of the current record-level validation mechanism. The geometric checks measure whether simulator-observable interaction records comply with predefined process constraints. This record-level interpretation was already discussed in Appendix A.4 (p. 19) of the original manuscript and is further clarified in Appendix A.4 (p. 19) and Appendix B.6.7 (pp. 37–38) of the revised manuscript.
To directly quantify the impact of geometric conditions on dynamic scores, we conducted an additional same-trajectory ablation. We disabled geometric checks on the original Table 4 dynamic trajectories while retaining answer correctness and all other applicable checks. Task instances, denominators, and task-family weights remained unchanged. The experimental procedure is described in Appendix B.6.7 (p. 38) of the revised manuscript, and the complete results are reported in Table 27 (p. 39).
| Model | SA (%) | JA (%) | TR (%) | BR (%) | IcS (%) | ImS (%) | CS (%) | MP (%) | Avg. (%) |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5.4 | 67.75 | 95.75 | 52.25 | 20.25 | 25.75 | 43.25 | 51.50 | 23.25 | 47.47 |
| Gemini-3.1-Pro | 92.00 | 77.00 | 56.75 | 92.00 | 29.50 | 86.25 | 43.00 | 52.00 | 66.06 |
| Claude-Sonnet-4.6 | 49.00 | 36.75 | 1.50 | 8.50 | 21.25 | 22.25 | 17.75 | 43.00 | 25.00 |
| Claude-Opus-4.6 | 97.75 | 97.75 | 81.75 | 61.00 | 55.25 | 96.00 | 57.00 | 43.25 | 73.72 |
| Grok-4 | 34.50 | 88.00 | 19.75 | 50.00 | 32.25 | 77.00 | 18.50 | 5.50 | 40.69 |
| GLM-5V | 75.00 | 3.00 | 1.25 | 0.50 | 5.25 | 2.50 | 0.50 | 0.50 | 11.06 |
| MiniMax-M2.7 | 12.25 | 48.00 | 2.00 | 0.25 | 1.50 | 16.25 | 0.75 | 21.50 | 12.81 |
| Qwen-Max | 4.50 | 1.00 | 0.75 | 0.25 | 3.75 | 17.00 | 0.75 | 0.50 | 3.56 |
The eight-family average scores and rankings under full validation and without geometric checks are summarized below:
| Model | Full avg. (%) | No-geometry avg. (%) | Full rank | No-geometry rank |
|---|---|---|---|---|
| GPT-5.4 | 26.31 | 47.47 | 2 | 3 |
| Gemini-3.1-Pro | 45.00 | 66.06 | 1 | 2 |
| Claude-Sonnet-4.6 | 5.66 | 25.00 | 5 | 5 |
| Claude-Opus-4.6 | 23.75 | 73.72 | 3 | 1 |
| Grok-4 | 11.34 | 40.69 | 4 | 4 |
| GLM-5V | 1.28 | 11.06 | 8 | 7 |
| MiniMax-M2.7 | 2.75 | 12.81 | 6 | 6 |
| Qwen-Max | 2.16 | 3.56 | 7 | 8 |
Regarding GPT-5.4's results on SA, where 18 episodes contained a correct answer but only one achieved full dynamic acceptance, the updated same-trajectory analysis provides a more precise interpretation. Among the same 50 recorded episodes, 18 contained at least one correct submission, but only one achieved joint success under full dynamic validation. The remaining 17 episodes did not satisfy the joint success criterion. One rejection mechanism observed in inspected trajectories involves reaching the correct position through multiple small-displacement adjustments, where the individual recorded drag gestures fail to satisfy the required displacement threshold.
Part 3
Question 2
Thank you for pointing out the inconsistency between the original rule description and the audit findings. Appendix A.4 (p. 19) of the original manuscript already specified that tile restoration supports both tap-based and drag-based swapping. However, the TR-specific description did not sufficiently explain the geometric checks applied in the experimental implementation. We have clarified this in Appendix A.4 (p. 20) of the revised manuscript. Full dynamic validation for TR includes applicable drag geometry checks, click geometry checks, and repeated-error checks. The drag and click geometry checks follow the corresponding conditions used for SA and IcS, respectively.
Both tapping and dragging are permitted interaction modes for TR. Therefore, an exchange should not be classified as task-illegal simply because the model uses dragging. Instead, the relevant question is whether the specific recorded operation satisfies the applicable dynamic constraints. Swap records and submission-consistency checks are used to verify whether the recorded swaps and final submission correspond to permitted state changes.
We have further refined the rejection analysis of the ten Claude-Opus-4.6 TR episodes examined in the original manuscript. Among these ten episodes, nine contained at least one correct submitted answer, while one never submitted a correct answer. For the first correct submission in each of the nine eligible episodes, the recorded rejection reasons are summarized below, as reported in Appendix B.6.5 and Table 20 (p. 34) of the revised manuscript:
| Recorded rejection reason | Count (N = 9) |
|---|---|
| Drag geometric non-compliance (A2) | 8 |
| Click geometric non-compliance (A3) | 1 |
| Repeated erroneous tile-pair swaps (A4) | 0 |
The revised analysis clarifies that all nine first correct submissions were recorded as failing geometric conditions: eight involved drag geometry, and one involved click geometry. None was recorded as a repeated erroneous tile-pair swap. Specifically, the repeated-error check concerns exchanging the same incorrect pair of tiles three consecutive times within a single attempt. The new ablation further supports this interpretation. When geometric checks are disabled, Claude-Opus-4.6's acceptance rate on the original TR trajectories increases from 0.00% to 81.75% (Table 27, p. 39 of the revised manuscript).
In addition, the explicit rule-disclosure experiment under the same agent framework shows that Claude-Opus-4.6's full dynamic acceptance on TR increases from 0/50 to 17/50, corresponding to an improvement from 0.00% to 34.00% (Table 22, p. 35 of the revised manuscript). These results demonstrate that the model can produce trajectories satisfying the configured rules under the current interaction interface.
Part 1
We sincerely thank the reviewer for the detailed assessment of our work. We appreciate your recognition of the diagnostic value of separating answer correctness from interaction acceptance, as well as your positive evaluation of the controlled experiments with shared instances, human calibration, validator failure analysis, state-recording comparisons, and ranking stability checks. We agree that the meaning of dynamic scores, the specific rejection rules, and the scope of our experimental conclusions require further clarification.
Weakness 1 and Question 1
First, we would like to clarify that full dynamic compliance in our benchmark includes predefined geometric constraints, together with the applicable interaction-history, state-transition, and consistency checks for each task family. Therefore, dynamic scores should be interpreted as completion acceptance rates under specified validation rules and agent execution configurations, rather than as general model capability scores independent of the validation conditions.
The distinction between answer correctness and full dynamic acceptance was described in Sections 3.3–3.4 (pp. 5–6) of the original manuscript and remains documented in Sections 3.3–3.4 (pp. 5–6) of the revised manuscript. It is important not to conflate different notions of validity. An operation may be permitted by the underlying task and may reach the correct answer while still failing to satisfy the geometric requirements of the full dynamic protocol. For tile restoration (TR), swapping two distinct tiles through the normal GUI is a permitted operation. Selecting the wrong pair of tiles constitutes a task-solving error, but does not, by itself, constitute an illegal state transition. Nevertheless, such swaps must also satisfy the applicable geometric constraints and repeated-error restrictions to pass full dynamic validation.
Independent verification may also assess geometric criteria. Its independence comes from reassessing the interaction using predefined rules and trajectory evidence rather than simply treating the validator's rejection label as evidence of a violation.
The "legal strategies" in Table 18 of the original manuscript refer to interaction strategies permitted by the underlying tasks. This does not imply that every trajectory produced by these strategies necessarily satisfies the complete dynamic protocol. In these controls, 1/20 small-step dragging traces and 3/20 center-clicking traces were rejected, as reported in Table 18 (p. 34) of the original manuscript and Table 21 (p. 35) of the revised manuscript.
The geometric checks do not treat the use of small-step dragging or center clicking as sufficient grounds for rejection. Instead, they evaluate whether the specific recorded interactions satisfy the predefined criteria. For example, rejection may occur when the displacements of individual drag gestures remain below the required threshold, or when the variance of click offsets relative to target centers falls below the specified threshold across multiple clicks. The corresponding rules are documented in Appendix A.4 (p. 19) of both the original and revised manuscripts. Therefore, interaction methods permitted by the underlying task must be distinguished from compliance with the complete dynamic protocol.
The motivation behind these rules is to incorporate behavioral interaction requirements into CAPTCHA evaluation. CAPTCHAs are designed to distinguish human users from automated programs. Accordingly, we aim to assess not only whether an agent can complete a CAPTCHA task, but also whether it can satisfy interaction constraints designed with human behavior in mind.
Related work has similarly investigated behavioral differences between humans and GUI agents using touch trajectories and temporal features, while separately evaluating behavioral human-likeness and task success [1]. This research supports the motivation for examining both whether a task is completed and how it is completed. Our geometric checks provide a limited and reproducible implementation of this motivation, and their results should be interpreted as compliance with predefined interaction rules.
Reference
[1] J. Zhu et al., “Passing the Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization,” in Proc. 32nd ACM SIGKDD Conf. Knowledge Discovery and Data Mining (KDD), vol. 2, 2026, pp. 10360–10371, doi: 10.1145/3770855.3817471.