acceptodds
Under review as a conference paper at ICLR 2027

HLL: Can Agents Cross Humanity’s Last Line of Verification?

Abstract

Multimodal agents are increasingly expected to operate graphical interfaces on behalf of users, yet real workflows often contain verification gates designed to constrain automation. Existing GUI-agent benchmarks emphasize navigation and task completion, while such verification steps are omitted. This leaves an important capability under-examined: whether an agent can couple visual understanding with grounded interaction, state tracking, and process-consistent execution under controlled verification constraints. To address this gap, we introduce **HLL**, a controlled benchmark for interactive CAPTCHA-style verification in an authorized local simulator. HLL spans ten task families and factorizes evaluation along three dimensions: intrinsic task difficulty, webpage distraction, and trace-conditioned interaction validation. Agents must produce semantically correct answers through grounded GUI actions, while dynamic variants additionally validate simulator-observable evidence. We evaluate eight multimodal agents in a closed-loop Android/Chrome environment using a common Mobile-Agent-v2 harness. Within-run audits show that correct submissions can still fail the configured dynamic rules. Supplementary controls also identify legal strategies rejected by these rules. A repair study recovers most tested geometric rejections through trace editing. HLL measures simulator-level process compliance, not behavioral authenticity.

Primary area:
datasets and benchmarks
Keywords:
Multimodal GUI Agents, Interactive Verification, CAPTCHA, Human-Computer Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DPbVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that CAPTCHA solving should be judged by whether the answer comes from a task-consistent interaction process beyond the final answer. To test this, the authors build HLL in a local Android/Chrome simulator: 10 task families and 33 evaluation cells (400 instances each), varied along three axes: webpage distraction, harder variants, and dynamic validation. Dynamic validation requires the recorded trace to pass rules on drags, clicks, repeated errors, and state transitions. The key result is a ranking reversal: Claude-Opus-4.6 leads the static setting (90.0%), while Gemini-3.1-Pro leads under dynamic validation (45.0%, vs. 26.3% for GPT-5.4 and 23.8% for Claude-Opus-4.6). From this, the authors conclude that answer-only evaluation can overstate agents’ verification ability. The appendix stress-tests the validators through human calibration, legality audits, and trace repair, with supplementary runs under AppAgent and UI-TARS. The authors position HLL as a simulator-level diagnostic.

Critical strengths
  1. Separating answer correctness from interaction acceptance is a useful diagnostic lens for GUI agents, and the controlled design makes this diagnosis credible: across ten task families, all models within a cell are evaluated on the same 400 seeded instances. The failure analysis in the main text is informative: weaker models mainly fail at perception and target identification, while stronger models more often reach the correct answer but fail at grounding, ordered interaction, or stateful execution. The authors also give a preliminary, single-model example of how the diagnosis can guide improvement: on 50 paired tile-restoration instances, explicit state recording solves 26 versus 19 for generic planning.

  2. Rather than assuming the validators are reliable, Section 4.2 tests whether the validators are reliable. Human participants pass in 239 of 240 trials, and every trace with removed or corrupted required records is rejected. The paper also reports the validators' failures: legal small-step and center-click strategies are wrongly rejected in 1 of 20 and 3 of 20 cases, and 1 of 10 illegal state transitions is missed. Section 4.5 adds an equal-budget comparison of answer-only and full validation, plus a trace-repair study, so readers can judge which conclusions depend on specific validator settings.

  3. The paper also checks the robustness of its rankings. Section 4.5 reports that Gemini-3.1-Pro keeps its lead when any single family is removed, while the order of GPT-5.4 and Claude-Opus-4.6 can flip. Gemini-3.1-Pro also stays highest when families are grouped by validation mechanism. The authors further rerun three models under AppAgent, add a UI-TARS run on the same instances, and state explicitly that these rankings describe complete agent configurations rather than isolated harness effects.

Critical weaknesses
  1. The dynamic layer is itself under-validated, so dynamic scores have no clear meaning. The paper's central finding, that static strength does not reliably transfer to dynamic acceptance, rests on these rules, yet their support is thin. The only human behavioral reference is three team members, and task legality is judged without an explicit criterion. The paper's own evidence is also in tension. Scripted small-step drags are treated as a legal strategy and rejected only once in 20 trials, yet 16 of the 43 rejected-but-correct episodes sampled for the repair study arise from agents reaching the correct position through several small drags, each below the displacement threshold. The paper does not say whether these 16 are legal. Combined with trace edits reversing 30 of 31 sampled geometric rejections once thresholds are known, this suggests the geometric rules check the form of the record rather than the legality of the interaction. How much of the dynamic score reflects genuine violations versus validator artifacts is therefore unknown. So is the meaning of results such as GPT-5.4 solving 18 slider instances under answer-only checking but only 1 under full validation at equal budgets.

  2. Each realism axis changes several things at once, so performance shifts cannot be attributed to the factor it names. The axes are meant to locate where, and under which conditions, the pipeline fails, which requires changing one variable at a time. Yet distraction changes layout and decoys together, and results on the same task swing both ways: on slider alignment, GPT-5.4 rises from 68.50% to 100%, while GLM-5V falls from 75.50% to 0%. The hard axis changes task content and budget together. Static and dynamic results come from separate rollouts under different budgets, with an equal-budget control for only one model on two families. The authors acknowledge in Section 4.4 that these conditions are composite, but then the design cannot answer the question it was built to answer.

  3. The main figures are of poor quality. Figures 1–3 are blurry and their small text is hard to read, which makes the benchmark overview difficult to follow. They give the impression of having been produced with generative tools without careful review. The authors should replace them with clear, publication-quality versions.

Questions for authors
  1. Two pieces of evidence don't line up. Section 4.2 treats scripted small-step drags as legal, rejecting them once in 20 trials. Yet 16 of the 43 sampled rejected-but-correct episodes come from agents reaching the target in several small drags, each below the displacement threshold. If such drags are legal, these are false rejections, and the 1-in-20 figure may understate real ones. If not, why does reaching the right position in steps violate the task? What exactly defines task legality?

  2. Claude-Opus-4.6 scores 96.75% on static tile restoration but 0/400 dynamically. The audit finds illegal interactions in 9 of 10 audited episodes, 8 of them "trajectory-related," yet the tile-restoration rules in Appendix A.4 include no trajectory check. Which rule rejects these submissions, and does Opus use a different input mode, such as drag instead of tap? If the harness's drag wrapper is the cause, this 0 reflects the environment, not the model.

  3. The only budget-controlled comparison of answer-only and full validation covers GPT-5.4 on two families: 18 versus 1 on slider and 10 versus 3 on tile restoration. AppAgent results, meanwhile, are reported only for the dynamic setting. Could you extend the comparison to a few representative models and families, and add static results under AppAgent? If the ranking reversal survives, the central claim becomes much stronger.

8 comments

We sincerely thank the reviewer for the detailed assessment of our work. We appreciate your recognition of the diagnostic value of separating answer correctness from interaction acceptance, as well as your positive evaluation of the controlled experiments with shared instances, human calibration, validator failure analysis, state-recording comparisons, and ranking stability checks. We agree that the meaning of dynamic scores, the specific rejection rules, and the scope of our experimental conclusions require further clarification.

Weakness 1 and Question 1

First, we would like to clarify that full dynamic compliance in our benchmark includes predefined geometric constraints, together with the applicable interaction-history, state-transition, and consistency checks for each task family. Therefore, dynamic scores should be interpreted as completion acceptance rates under specified validation rules and agent execution configurations, rather than as general model capability scores independent of the validation conditions.

The distinction between answer correctness and full dynamic acceptance was described in Sections 3.3–3.4 (pp. 5–6) of the original manuscript and remains documented in Sections 3.3–3.4 (pp. 5–6) of the revised manuscript. It is important not to conflate different notions of validity. An operation may be permitted by the underlying task and may reach the correct answer while still failing to satisfy the geometric requirements of the full dynamic protocol. For tile restoration (TR), swapping two distinct tiles through the normal GUI is a permitted operation. Selecting the wrong pair of tiles constitutes a task-solving error, but does not, by itself, constitute an illegal state transition. Nevertheless, such swaps must also satisfy the applicable geometric constraints and repeated-error restrictions to pass full dynamic validation.

Independent verification may also assess geometric criteria. Its independence comes from reassessing the interaction using predefined rules and trajectory evidence rather than simply treating the validator's rejection label as evidence of a violation.

The "legal strategies" in Table 18 of the original manuscript refer to interaction strategies permitted by the underlying tasks. This does not imply that every trajectory produced by these strategies necessarily satisfies the complete dynamic protocol. In these controls, 1/20 small-step dragging traces and 3/20 center-clicking traces were rejected, as reported in Table 18 (p. 34) of the original manuscript and Table 21 (p. 35) of the revised manuscript.

The geometric checks do not treat the use of small-step dragging or center clicking as sufficient grounds for rejection. Instead, they evaluate whether the specific recorded interactions satisfy the predefined criteria. For example, rejection may occur when the displacements of individual drag gestures remain below the required threshold, or when the variance of click offsets relative to target centers falls below the specified threshold across multiple clicks. The corresponding rules are documented in Appendix A.4 (p. 19) of both the original and revised manuscripts. Therefore, interaction methods permitted by the underlying task must be distinguished from compliance with the complete dynamic protocol.

The motivation behind these rules is to incorporate behavioral interaction requirements into CAPTCHA evaluation. CAPTCHAs are designed to distinguish human users from automated programs. Accordingly, we aim to assess not only whether an agent can complete a CAPTCHA task, but also whether it can satisfy interaction constraints designed with human behavior in mind.

Related work has similarly investigated behavioral differences between humans and GUI agents using touch trajectories and temporal features, while separately evaluating behavioral human-likeness and task success [1]. This research supports the motivation for examining both whether a task is completed and how it is completed. Our geometric checks provide a limited and reproducible implementation of this motivation, and their results should be interpreted as compliance with predefined interaction rules.

Reference

[1] J. Zhu et al., “Passing the Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization,” in Proc. 32nd ACM SIGKDD Conf. Knowledge Discovery and Data Mining (KDD), vol. 2, 2026, pp. 10360–10371, doi: 10.1145/3770855.3817471.

Regarding the 16 small-step dragging cases in the repair study, we have reexamined these cases and confirmed that they reached the correct final position, but the recorded sequence of short drag gestures did not satisfy the predefined displacement threshold. The rejections were therefore triggered by this geometric condition. Accordingly, these cases should be explicitly described as "answer-correct but non-compliant with the geometric requirements of the full dynamic protocol." The relevant rules are documented in Appendix A.4 (p. 19) of both the original and revised manuscripts, while the repair mechanisms are described in Appendix B.6.7 (p. 35) of the original manuscript and Appendix B.6.7 (p. 37) of the revised manuscript.

Table 20 (p. 35) of the original manuscript and Table 26 (p. 38) of the revised manuscript report that, while preserving the submitted answer and final position, trace editing recovered 22/31 geometric rejections using default parameters and 30/31 using predetermined threshold-aware parameters. Here, "recovery" means that the modified record passes the predefined dynamic checks. It does not imply that the original record satisfied those checks.

We acknowledge that these results reveal a limitation of the current record-level validation mechanism. The geometric checks measure whether simulator-observable interaction records comply with predefined process constraints. This record-level interpretation was already discussed in Appendix A.4 (p. 19) of the original manuscript and is further clarified in Appendix A.4 (p. 19) and Appendix B.6.7 (pp. 37–38) of the revised manuscript.

To directly quantify the impact of geometric conditions on dynamic scores, we conducted an additional same-trajectory ablation. We disabled geometric checks on the original Table 4 dynamic trajectories while retaining answer correctness and all other applicable checks. Task instances, denominators, and task-family weights remained unchanged. The experimental procedure is described in Appendix B.6.7 (p. 38) of the revised manuscript, and the complete results are reported in Table 27 (p. 39).

ModelSA (%)JA (%)TR (%)BR (%)IcS (%)ImS (%)CS (%)MP (%)Avg. (%)
GPT-5.467.7595.7552.2520.2525.7543.2551.5023.2547.47
Gemini-3.1-Pro92.0077.0056.7592.0029.5086.2543.0052.0066.06
Claude-Sonnet-4.649.0036.751.508.5021.2522.2517.7543.0025.00
Claude-Opus-4.697.7597.7581.7561.0055.2596.0057.0043.2573.72
Grok-434.5088.0019.7550.0032.2577.0018.505.5040.69
GLM-5V75.003.001.250.505.252.500.500.5011.06
MiniMax-M2.712.2548.002.000.251.5016.250.7521.5012.81
Qwen-Max4.501.000.750.253.7517.000.750.503.56

The eight-family average scores and rankings under full validation and without geometric checks are summarized below:

ModelFull avg. (%)No-geometry avg. (%)Full rankNo-geometry rank
GPT-5.426.3147.4723
Gemini-3.1-Pro45.0066.0612
Claude-Sonnet-4.65.6625.0055
Claude-Opus-4.623.7573.7231
Grok-411.3440.6944
GLM-5V1.2811.0687
MiniMax-M2.72.7512.8166
Qwen-Max2.163.5678

Regarding GPT-5.4's results on SA, where 18 episodes contained a correct answer but only one achieved full dynamic acceptance, the updated same-trajectory analysis provides a more precise interpretation. Among the same 50 recorded episodes, 18 contained at least one correct submission, but only one achieved joint success under full dynamic validation. The remaining 17 episodes did not satisfy the joint success criterion. One rejection mechanism observed in inspected trajectories involves reaching the correct position through multiple small-displacement adjustments, where the individual recorded drag gestures fail to satisfy the required displacement threshold.

Question 2

Thank you for pointing out the inconsistency between the original rule description and the audit findings. Appendix A.4 (p. 19) of the original manuscript already specified that tile restoration supports both tap-based and drag-based swapping. However, the TR-specific description did not sufficiently explain the geometric checks applied in the experimental implementation. We have clarified this in Appendix A.4 (p. 20) of the revised manuscript. Full dynamic validation for TR includes applicable drag geometry checks, click geometry checks, and repeated-error checks. The drag and click geometry checks follow the corresponding conditions used for SA and IcS, respectively.

Both tapping and dragging are permitted interaction modes for TR. Therefore, an exchange should not be classified as task-illegal simply because the model uses dragging. Instead, the relevant question is whether the specific recorded operation satisfies the applicable dynamic constraints. Swap records and submission-consistency checks are used to verify whether the recorded swaps and final submission correspond to permitted state changes.

We have further refined the rejection analysis of the ten Claude-Opus-4.6 TR episodes examined in the original manuscript. Among these ten episodes, nine contained at least one correct submitted answer, while one never submitted a correct answer. For the first correct submission in each of the nine eligible episodes, the recorded rejection reasons are summarized below, as reported in Appendix B.6.5 and Table 20 (p. 34) of the revised manuscript:

Recorded rejection reasonCount (N = 9)
Drag geometric non-compliance (A2)8
Click geometric non-compliance (A3)1
Repeated erroneous tile-pair swaps (A4)0

The revised analysis clarifies that all nine first correct submissions were recorded as failing geometric conditions: eight involved drag geometry, and one involved click geometry. None was recorded as a repeated erroneous tile-pair swap. Specifically, the repeated-error check concerns exchanging the same incorrect pair of tiles three consecutive times within a single attempt. The new ablation further supports this interpretation. When geometric checks are disabled, Claude-Opus-4.6's acceptance rate on the original TR trajectories increases from 0.00% to 81.75% (Table 27, p. 39 of the revised manuscript).

In addition, the explicit rule-disclosure experiment under the same agent framework shows that Claude-Opus-4.6's full dynamic acceptance on TR increases from 0/50 to 17/50, corresponding to an improvement from 0.00% to 34.00% (Table 22, p. 35 of the revised manuscript). These results demonstrate that the model can produce trajectories satisfying the configured rules under the current interaction interface.

Reviewer_9Qr8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HLL evaluates multimodal GUI agents on ten CAPTCHA-style task families in a local simulator. It separates answer correctness from acceptance under interaction rules and tests eight agents across difficulty, distraction, and dynamic-validation settings. The results reveal execution failures that answer-only scoring misses, while additional audits also expose limitations in the validators.

Critical strengths
  1. Ten task families and 400 shared instances per setting provide a substantial, controlled evaluation.
  2. The human calibration, independent legality audits, and trace-repair experiments examine both agent errors and evaluator errors.
Critical weaknesses
  1. Some legal actions, including center clicks and small-step drags, are rejected by the dynamic checks in Table 18. This makes it unclear how much low scores reflect poor execution versus restrictions imposed by the validator.
  2. GPT and Gemini switch rankings under AppAgent in Table 13. The author acknowledges this dependence on the agent setup, but does not fully explain what drives it, limiting the conclusions about specific capability gaps.
Questions for authors
  1. Can you report independently verified legal-completion rates and show how scores and rankings change when the geometric checks are removed?
  2. Can you extend the equal-budget comparisons to more models and tasks, and distinguish agent decision errors from execution or logging issues?
7 comments

We sincerely thank the reviewer for the positive assessment of our work, particularly for recognizing the controlled evaluation enabled by ten task families and shared instances, as well as our human calibration, independent legality audits, and trace-repair experiments. We agree that it is important to further distinguish agent performance from the effects of validation rules and to provide a more specific explanation of ranking changes across agent frameworks.

Weakness 1 and Question 1

First, we would like to clarify the specific meaning of the geometric checks. These rules do not automatically classify all center clicks or small-step drags as violations. The click-based checks examine whether target-relative offsets across multiple targets are excessively consistent, while the drag-based checks examine recorded displacement and trajectory continuity. These rules were described in Appendix A.4 (p. 19) of the original manuscript and remain documented in Appendix A.4 (p. 19) of the revised manuscript.

Geometric checks are part of the full dynamic validity criterion defined in our benchmark. They do not change how final-answer correctness is determined, but they affect the joint acceptance rate, which requires both a correct answer and satisfaction of all applicable dynamic checks. Therefore, dynamic scores should be interpreted as completion acceptance rates under specified validation rules and agent execution configurations, rather than as general capability scores independent of the validation conditions.

The motivation behind these rules is to incorporate behavioral interaction requirements into CAPTCHA evaluation. CAPTCHAs are designed to distinguish human users from automated programs. Accordingly, we aim to evaluate not only whether an agent can complete a CAPTCHA task, but also whether it can satisfy interaction constraints designed with human behavior in mind. Related work has similarly investigated behavioral differences between humans and GUI agents using touch trajectories and temporal features, while separately evaluating behavioral human-likeness and task success [1]. This research supports the motivation for examining both whether a task is completed and how it is completed. Our geometric checks provide a limited and reproducible implementation of this motivation, and their results should be interpreted as compliance with predefined interaction rules.

We also acknowledge the limitation highlighted by the reviewer. In Table 18 (p. 34) of the original manuscript and Table 21 (p. 35) of the revised manuscript, 1/20 independently confirmed task-legal small-step dragging traces and 3/20 task-legal center-clicking traces were rejected. These results demonstrate that task-level legality and full dynamic acceptance can indeed disagree. Meanwhile, in our human calibration, 239 of 240 interactions with correct answers passed full dynamic validation, corresponding to an observed rejection rate of 0.42% (original manuscript: Appendix B.2 and Table 8, p. 23; revised manuscript: Appendix B.2, p. 23, and Table 9, p. 24).

Regarding independent verification, we report two complementary sources of evidence. First, Appendix B.6.5 (p. 33) of both the original and revised manuscripts reports an independent audit of 19 cases in which the first correct submission was rejected, comprising five SA, five IcS, four TR, and five MP cases. The audit assessed task legality using the task rules and available page and action evidence, rather than treating the validator's rejection label as evidence of illegality. All 19 cases were judged task-illegal, with no cases classified as legal or indeterminate. No inconsistency between recorded and observed actions was identified in these inspected cases.

Second, the original manuscript already reported a TR comparison using "a correct answer with independently confirmed legal interaction" as the success criterion in Appendix B.6.6 and Table 19 (p. 34). The corresponding results are presented in Appendix B.6.6 (p. 35) and Table 23 (p. 36) of the revised manuscript.

Under the GPT-5.4 and Mobile-Agent-v2 configuration, the two planning conditions were evaluated through separate online runs on the same 50 instances with matched budgets. The independently verified legal-completion rates are as follows:

FrameworkModelTaskPlanning conditionNIndependently verified legal completion (%)
Mobile-Agent-v2GPT-5.4TRGeneric planning5038.00
Mobile-Agent-v2GPT-5.4TRExplicit state recording5052.00

References

[1] J. Zhu et al., “Passing the Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization,” in Proc. 32nd ACM SIGKDD Conf. Knowledge Discovery and Data Mining (KDD), vol. 2, 2026, pp. 10360–10371, doi: 10.1145/3770855.3817471.

To address the reviewer's question about removing geometric checks, we conducted an additional offline rescoring experiment using the same dynamic trajectories from Table 4 of the original manuscript. We disabled only the geometric predicates while retaining answer correctness and all other applicable checks. Task instances, denominators, episode-level success definitions, and task-family weights remained unchanged.

For SA, JA, ImS, and IcS, the primary dynamic constraints are geometric checks. For CS, MP, TR, and BR, the applicable history- and state-based checks were retained during the ablation. The experimental procedure is described in Appendix B.6.7 (p. 38) of the revised manuscript, and the complete results are reported in Table 27 (p. 39).

ModelSA (%)JA (%)TR (%)BR (%)IcS (%)ImS (%)CS (%)MP (%)Avg. (%)
GPT-5.467.7595.7552.2520.2525.7543.2551.5023.2547.47
Gemini-3.1-Pro92.0077.0056.7592.0029.5086.2543.0052.0066.06
Claude-Sonnet-4.649.0036.751.508.5021.2522.2517.7543.0025.00
Claude-Opus-4.697.7597.7581.7561.0055.2596.0057.0043.2573.72
Grok-434.5088.0019.7550.0032.2577.0018.505.5040.69
GLM-5V75.003.001.250.505.252.500.500.5011.06
MiniMax-M2.712.2548.002.000.251.5016.250.7521.5012.81
Qwen-Max4.501.000.750.253.7517.000.750.503.56

The eight-family average scores and rankings under full validation and without geometric checks are summarized below:

ModelFull avg. (%)No-geometry avg. (%)Full rankNo-geometry rank
GPT-5.426.3147.4723
Gemini-3.1-Pro45.0066.0612
Claude-Sonnet-4.65.6625.0055
Claude-Opus-4.623.7573.7231
Grok-411.3440.6944
GLM-5V1.2811.0687
MiniMax-M2.72.7512.8166
Qwen-Max2.163.5678

The results show that Claude-Opus-4.6's average acceptance rate increases from 23.75% to 73.72%, indicating that its performance under full dynamic validation is substantially affected by geometric constraints. Accordingly, we interpret full dynamic scores as acceptance rates under predefined validation rules and separately quantify the impact of geometric conditions through this ablation.

Weakness 2 and Question 2

Thank you for pointing out that simply acknowledging the dependence of model rankings on agent configurations does not sufficiently explain what drives these changes. We have added static evaluations under AppAgent and conducted a task-level contribution analysis together with matched-budget, same-trajectory comparisons to further distinguish differences in answer correctness from those in dynamic acceptance.

First, the overall ranking reversal in the original Table 13 is primarily concentrated in TR and BR, rather than reflecting consistent improvements across all tasks. The relevant results are reported in Table 4 (p. 8) and Appendix B.6.2, Table 13 (p. 31) of the original manuscript, and in Table 4 (p. 9) and Appendix B.6.2, Table 15 (p. 32) of the revised manuscript.

Under Mobile-Agent-v2, the dynamic average scores of GPT-5.4 and Gemini-3.1-Pro are 26.31% and 45.00%, respectively. Under AppAgent, they are 34.34% and 29.44%, respectively. By decomposing the changes in the GPT–Gemini score difference across the eight task families, we find that TR contributes 7.34 percentage points and BR contributes 9.94 percentage points. Together, these two families account for approximately 73.2% of the total 23.59-percentage-point change in GPT's relative performance against Gemini. Specifically, on TR, the GPT/Gemini success rates change from 50.25%/49.75% under Mobile-Agent-v2 to 62.25%/3.00% under AppAgent. On BR, they change from 18.75%/90.25% to 39.75%/31.75%.

Second, the newly added AppAgent static evaluation shows that differences between the two models are already present at the level of answer correctness. We supplemented the evaluation with AppAgent static results for three models across all ten base task families, using 50 instances per model and task family. These results are reported in Appendix B.6.2, Table 14 (p. 31) of the revised manuscript. Under the static condition, success is determined solely by answer correctness. The results are as follows:

ModelText (%)SA (%)ImS (%)JA (%)LA (%)CS (%)MP (%)IcS (%)TR (%)BR (%)
GPT-5.4100.0058.0062.0082.0056.0074.0082.0028.0070.0062.00
Gemini-3.1-Pro98.0054.0068.0026.0038.0042.0086.0022.0036.0086.00
Claude-Opus-4.6100.0058.0062.0064.0070.0074.0082.0042.0068.0038.00

Across the eight task families corresponding to the dynamic evaluation, excluding Text and LA, the AppAgent static averages are 64.75% for GPT-5.4 and 52.50% for Gemini-3.1-Pro. For descriptive comparison, the corresponding static averages in the main Mobile-Agent-v2 experiments are 65.6% and 71.1%, respectively (Static Avg. column in Table 4, p. 9 of the revised manuscript). These results indicate that configuration-dependent differences in model ordering are also present when evaluation is based solely on answer correctness.

The matched-budget, same-trajectory TR comparison further shows that full dynamic validation amplifies the existing answer-correctness gap between GPT and Gemini under AppAgent.

In the new experiment, we apply the answer-correctness and full dynamic validation criteria to the same trajectories from each online run. Each model–framework–task configuration uses 50 instances under a unified budget. The results are reported in Table 24 (p. 36) of the revised manuscript. The TR results are summarized below:

FrameworkModelAnswer-only A (%)Full validation B (%)A − B (pp)
Mobile-Agent-v2GPT-5.450.0046.004.00
Mobile-Agent-v2Gemini-3.1-Pro52.0048.004.00
AppAgentGPT-5.470.0064.006.00
AppAgentGemini-3.1-Pro36.006.0030.00

Under Mobile-Agent-v2, the two models have similar answer-correctness rates, and the reductions associated with full validation are both 4 percentage points. Under AppAgent, GPT's advantage over Gemini is already 34 percentage points in answer correctness, and this advantage increases to 58 percentage points under full dynamic acceptance. Specifically, the difference between answer correctness and full acceptance is 6 percentage points for GPT, compared with 30 percentage points for Gemini.

Question 2

The original validation comparison in Appendix B.6.6 and Table 19 (p. 34) covered only GPT-5.4 on two task families. We have extended this comparison to three models, four task families, and two agent frameworks. The complete results are reported in Appendix B.6.6, Table 24 (p. 36) of the revised manuscript.

Each model–framework–task configuration contains 50 instances. We use a unified budget of at most 24 GUI actions and three failed verifications per attempt, up to three attempts per episode, and a total timeout of 2,400 seconds per episode. We apply two scoring criteria to the same recorded trajectories from a single online execution:

  • Criterion A (answer-only): An episode is successful if it contains at least one correct submitted answer.
  • Criterion B (full validation): An episode is successful if at least one submission is both correct and accepted by all applicable dynamic checks.
FrameworkModelTaskAnswer-only A (%)Full validation B (%)B − A (pp)
Mobile-Agent-v2GPT-5.4SA36.002.00−34.00
Mobile-Agent-v2GPT-5.4IcS28.008.00−20.00
Mobile-Agent-v2GPT-5.4TR50.0046.00−4.00
Mobile-Agent-v2GPT-5.4MP92.0024.00−68.00
Mobile-Agent-v2Gemini-3.1-ProSA90.0052.00−38.00
Mobile-Agent-v2Gemini-3.1-ProIcS32.0024.00−8.00
Mobile-Agent-v2Gemini-3.1-ProTR52.0048.00−4.00
Mobile-Agent-v2Gemini-3.1-ProMP76.0056.00−20.00
Mobile-Agent-v2Claude-Opus-4.6SA96.008.00−88.00
Mobile-Agent-v2Claude-Opus-4.6IcS50.0026.00−24.00
Mobile-Agent-v2Claude-Opus-4.6TR92.004.00−88.00
Mobile-Agent-v2Claude-Opus-4.6MP96.0042.00−54.00
AppAgentGPT-5.4SA58.000.00−58.00
AppAgentGPT-5.4IcS28.0010.00−18.00
AppAgentGPT-5.4TR70.0064.00−6.00
AppAgentGPT-5.4MP82.0048.00−34.00
AppAgentGemini-3.1-ProSA54.0052.00−2.00
AppAgentGemini-3.1-ProIcS22.0016.00−6.00
AppAgentGemini-3.1-ProTR36.006.00−30.00
AppAgentGemini-3.1-ProMP86.0056.00−30.00
AppAgentClaude-Opus-4.6SA58.006.00−52.00
AppAgentClaude-Opus-4.6IcS42.0016.00−26.00
AppAgentClaude-Opus-4.6TR66.006.00−60.00
AppAgentClaude-Opus-4.6MP82.0022.00−60.00

Across all 24 configurations, full dynamic acceptance rates are lower than answer-correctness rates, with observed gaps ranging from 2 to 88 percentage points.

Reviewer_oY9GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HLL, a benchmark for interactive CAPTCHA tasks. Besides checking the final answer, it also checks action traces and state changes. The authors evaluate several models and agent frameworks under different settings.

Critical strengths
  1. Separating answer correctness from interaction validation is useful. It gives more information than only reporting whether the task is completed.
  2. The authors also examine false rejections and trace repair. I appreciate that these limitations are reported openly.
Critical weaknesses
  1. Some valid actions, such as clicking exactly at the center or dragging in several small steps, can still be rejected (Table 18). So it is not very clear how much the low score reflects model capability, and how much comes from these extra rules.

  2. Opus drops from 96.75% in the static setting to 0% in the dynamic setting on tile restoration. This is a large change. The authors checked ten examples, but the explanation is still quite general. I would like to understand which step actually went wrong.

  3. The static and dynamic experiments use different budgets. There is a matched-budget comparison in appendix, but it only covers GPT-5.4 on two task families. Whether the same finding holds for other models and tasks is still unclear.

  4. Some details are missing for reproduction, such as the actual prompts, where the images come from, and the size of the image pool. These details also matter for judging the diversity of the tasks.

Questions for authors
  1. Are the restrictions on center clicks and small-step dragging explained to the agent? What capability are these rules intended to measure?
  2. Could you show representative Opus failure traces on tile restoration, including the action, page state and recorded event where the violation happens?
  3. Could you add a matched-budget comparison on another model or task family to check whether the finding is stable?
  4. Could you provide the actual prompts and more details about the image sources and pool size?
5 comments

We sincerely thank the reviewer for the positive assessment of our work. We appreciate your recognition that distinguishing answer correctness from interaction validation provides richer diagnostic information than task completion rates alone, as well as your acknowledgment of our transparent analysis of false rejections and the limitations of trace repair. In the revision, we retain and further clarify these limitations while adding experiments on explicit rule disclosure, matched-budget comparisons across models and task families, and additional details about prompts and image sources.

Weakness 1

Thank you for raising these important concerns. First, we clarify that geometric checks are part of the full dynamic validity criterion defined in our benchmark. They do not change how final-answer correctness is determined, but they affect the joint acceptance rate, which requires both a correct answer and satisfaction of all applicable dynamic checks. Therefore, dynamic scores should be interpreted as completion acceptance rates under specified validation rules and agent execution configurations, rather than as general capability scores independent of the validation conditions.

The original manuscript distinguishes answer correctness from dynamic acceptance in Section 3.3 (p. 5) and Section 3.4 (pp. 5–6). The revised manuscript further clarifies the distinction between full dynamic acceptance and independently assessed task legality in Section 3.3 (p. 5) and Section 3.4 (p. 6).

Regarding your observation that legal strategies can be rejected, the original manuscript explicitly reports rejections involving small-step dragging and center clicking in Table 18 (p. 34). These results are retained in Table 21 (p. 35) of the revised manuscript.

Our geometric rules were calibrated using human interactions. Among 240 human trials with correct answers, 239 passed full dynamic validation, corresponding to an observed false-rejection rate of 0.42% (original manuscript: Appendix B.2 and Table 8, p. 23; revised manuscript: Appendix B.2, p. 23, and Table 9, p. 24). This result supports a high acceptance rate for normal human interactions within the tested calibration sample.

It is important to distinguish answer correctness from full dynamic acceptance. Geometric rules do not modify the answer-correctness criterion. Instead, they constitute an additional component of dynamic validity and jointly determine final acceptance with answer correctness. Consequently, an interaction that produces the correct answer may still be rejected if its recorded behavior does not satisfy the geometric constraints. We explicitly retain and report such cases.

The motivation behind these rules is to incorporate behavioral interaction requirements into CAPTCHA evaluation. CAPTCHAs are designed to distinguish human users from automated programs. Accordingly, we aim to evaluate not only whether an agent can solve a CAPTCHA, but also whether it can complete the task while satisfying interaction constraints designed with human behavior in mind.

Related work has similarly investigated behavioral differences between humans and GUI agents using touch trajectories and temporal features, while separately evaluating behavioral human-likeness and task success [1]. This research supports the motivation for examining both whether a task is completed and how it is completed. Our geometric checks provide a limited and reproducible implementation of this motivation, and their results should be interpreted as compliance with predefined interaction rules.

[1] J. Zhu et al., “Passing the Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization,” in Proc. 32nd ACM SIGKDD Conf. Knowledge Discovery and Data Mining (KDD), vol. 2, 2026, pp. 10360–10371, doi: 10.1145/3770855.3817471.

To quantify the extent to which geometric rules contribute to low dynamic scores, we additionally rescored the same trajectories from the original dynamic experiments with geometric checks disabled, while retaining answer correctness and all other applicable checks. Task instances, execution trajectories, denominators, and task-family weights remained unchanged. The experimental procedure is described in Appendix B.6.7 (p. 38) of the revised manuscript, with per-family results reported in Table 27 (p. 39). The acceptance rates after removing geometric checks are as follows:

ModelSAJATRBRIcSImSCSMPAvg. (%)
GPT-5.467.7595.7552.2520.2525.7543.2551.5023.2547.47
Gemini-3.1-Pro92.0077.0056.7592.0029.5086.2543.0052.0066.06
Claude-Sonnet-4.649.0036.751.508.5021.2522.2517.7543.0025.00
Claude-Opus-4.697.7597.7581.7561.0055.2596.0057.0043.2573.72
Grok-434.5088.0019.7550.0032.2577.0018.505.5040.69
GLM-5V75.003.001.250.505.252.500.500.5011.06
MiniMax-M2.712.2548.002.000.251.5016.250.7521.5012.81
Qwen-Max4.501.000.750.253.7517.000.750.503.56

The eight-family averages and model rankings under full validation and without geometric checks are summarized below:

ModelFull validation (%)Without geometric checks (%)Full rankNo-geometry rank
GPT-5.426.3147.4723
Gemini-3.1-Pro45.0066.0612
Claude-Sonnet-4.65.6625.0055
Claude-Opus-4.623.7573.7231
Grok-411.3440.6944
GLM-5V1.2811.0687
MiniMax-M2.72.7512.8166
Qwen-Max2.163.5678

The results show that Claude-Opus-4.6's average acceptance rate increases from 23.75% to 73.72%, indicating that geometric constraints substantially affect its performance under full dynamic validation.

Accordingly, we explicitly interpret full dynamic scores as acceptance rates under predefined validation rules and separately quantify the contribution of geometric conditions through this ablation.

Regarding whether the agent was informed of these rules, the original experiments included a general instruction to interact in a deliberate, human-like manner, but did not explicitly disclose specific geometric rules or numerical thresholds. The original dynamic instruction is provided in Appendix C.2 (p. 44) of the revised manuscript. To further investigate this issue, we conducted an explicit rule-disclosure experiment covering three models and four task families, with 50 instances for each model–task–prompt condition. Both prompt conditions used the same budgets. The complete results are reported in Appendix B.6.6, “Explicit Rule Disclosure,” and Table 22 (p. 35) of the revised manuscript. The following table compares full dynamic acceptance and answer correctness before and after explicit rule disclosure:

ModelTaskOriginal full (%)Disclosed full (%)Original answer (%)Disclosed answer (%)
GPT-5.4SA4.0022.0036.0026.00
GPT-5.4IcS12.0016.0028.0018.00
Gemini-3.1-ProSA52.0082.0092.0088.00
Gemini-3.1-ProIcS18.0020.0030.0022.00
Claude-Opus-4.6SA12.0076.0096.0082.00
Claude-Opus-4.6IcS22.0026.0050.0034.00
Claude-Opus-4.6TR0.0034.0092.0078.00
GPT-5.4TR48.0048.0050.0048.00
Gemini-3.1-ProTR54.0054.0056.0056.00
Claude-Opus-4.6MP42.0040.0094.0078.00
GPT-5.4MP22.0022.0092.0080.00
Gemini-3.1-ProMP48.0050.0076.0064.00

For slider alignment (SA), explicitly disclosing the rules leads to substantial improvements in full dynamic acceptance. The improvements are smaller for icon sequence selection (IcS) and missing-patch selection (MP). For tile restoration (TR), Claude-Opus-4.6 improves, whereas GPT-5.4 and Gemini-3.1-Pro show no improvement. At the same time, rule disclosure does not always preserve answer correctness. For example, on IcS, the numbers of episodes with correct submitted answers decrease from 14, 15, and 25 to 9, 11, and 17 for GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4.6, respectively. These results indicate that explicit rule disclosure can improve dynamic acceptance in certain configurations, but it does not eliminate the gap across all tasks and may be accompanied by a reduction in answer correctness.

Weakness 2

Thank you for pointing out that our original analysis lacked sufficient detail. Tile restoration (TR) supports both tap-based and drag-based tile swapping. This is explicitly stated in the original manuscript under the “Tile restoration” entry in Appendix A.4 (p. 19), which specifies that a swap can be performed through “either a tap-based or drag-based flow.” In the revised manuscript, we further clarify in the corresponding entry in Appendix A.4 (p. 20) that full dynamic validation for TR includes not only swap-state checks but also applicable drag geometry checks, click geometry checks, and repeated-error checks.

The original manuscript reported an audit of ten randomly selected Claude-Opus-4.6 dynamic TR episodes in Appendix B.6.5, “Focused Audit of Claude-Opus-4.6 on Tile Restoration” (p. 33). We have now supplemented this analysis with a more specific breakdown of the recorded rejection reasons in Appendix B.6.5 and Table 20 (p. 34) of the revised manuscript.

Among the ten audited episodes, nine contained at least one correct submitted answer, while one never submitted a correct answer. All ten failed full dynamic acceptance. For the first correct submission in each of the nine eligible episodes, the recorded rejection reasons are summarized below:

Recorded rejection reasonCount (N = 9)
Drag geometric non-compliance (A2)8
Click geometric non-compliance (A3)1
Repeated erroneous tile-pair swaps (A4)0

These results indicate that, among the audited first correct submissions, the primary failure mechanism was non-compliance with geometric checks in the recorded interactions, rather than a failure to submit the correct tile-restoration answer.

Weakness 3

The original validation comparison in Appendix B.6.6 and Table 19 (p. 34) covered only GPT-5.4 on two task families, SA and TR. We have extended this analysis to three models (GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4.6), four task families (SA, IcS, TR, and MP), and two agent frameworks (Mobile-Agent-v2 and AppAgent). The complete results are reported in Appendix B.6.6 and Table 24 (p. 36) of the revised manuscript.

Each model–framework–task configuration contains 50 instances. We use a unified budget of at most 24 GUI actions and three failed verifications per attempt, up to three attempts per episode, and a total timeout of 2,400 seconds per episode. We apply two evaluation criteria to the same recorded trajectories from a single online execution:

  • Criterion A (answer-only): Success requires a correct submitted answer.
  • Criterion B (full validation): Success requires a correct submitted answer and acceptance by all applicable dynamic checks.

The results are as follows:

FrameworkModelTaskAnswer-only A (%)Full validation B (%)B − A (pp)
Mobile-Agent-v2GPT-5.4SA36.002.00−34.00
Mobile-Agent-v2GPT-5.4IcS28.008.00−20.00
Mobile-Agent-v2GPT-5.4TR50.0046.00−4.00
Mobile-Agent-v2GPT-5.4MP92.0024.00−68.00
Mobile-Agent-v2Gemini-3.1-ProSA90.0052.00−38.00
Mobile-Agent-v2Gemini-3.1-ProIcS32.0024.00−8.00
Mobile-Agent-v2Gemini-3.1-ProTR52.0048.00−4.00
Mobile-Agent-v2Gemini-3.1-ProMP76.0056.00−20.00
Mobile-Agent-v2Claude-Opus-4.6SA96.008.00−88.00
Mobile-Agent-v2Claude-Opus-4.6IcS50.0026.00−24.00
Mobile-Agent-v2Claude-Opus-4.6TR92.004.00−88.00
Mobile-Agent-v2Claude-Opus-4.6MP96.0042.00−54.00
AppAgentGPT-5.4SA58.000.00−58.00
AppAgentGPT-5.4IcS28.0010.00−18.00
AppAgentGPT-5.4TR70.0064.00−6.00
AppAgentGPT-5.4MP82.0048.00−34.00
AppAgentGemini-3.1-ProSA54.0052.00−2.00
AppAgentGemini-3.1-ProIcS22.0016.00−6.00
AppAgentGemini-3.1-ProTR36.006.00−30.00
AppAgentGemini-3.1-ProMP86.0056.00−30.00
AppAgentClaude-Opus-4.6SA58.006.00−52.00
AppAgentClaude-Opus-4.6IcS42.0016.00−26.00
AppAgentClaude-Opus-4.6TR66.006.00−60.00
AppAgentClaude-Opus-4.6MP82.0022.00−60.00

Across all 24 configurations, full dynamic acceptance rates are lower than answer correctness rates, with observed gaps ranging from 2 to 88 percentage points. Because both evaluation criteria are applied to the same recorded trajectories, these differences cannot be attributed to unequal action or time budgets between static and dynamic experiments. These results demonstrate that obtaining a correct answer does not guarantee acceptance under full interaction validation, and that this phenomenon extends across multiple models, task families, and agent frameworks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.