- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces CaptchaArena, a dataset comprising 50K instances covering 20 captcha categories and 5 interaction modes. It provides execution-verified screenshot-action trajectories, with 46K entries including reasoning annotations and pixel-level masks for irregular click regions. Using Qwen3.5-9B as the backbone, the authors perform SFT with 12K training trajectories followed by GRPO based on environmental feedback to obtain a unified model, CaptchaAgent. Pass@1 accuracy improves from 11.4% (base model) to 70.5% (SFT), reaching 71.7% after RL. On external benchmarks Open CaptchaWorld and Halligan, it achieves 51.0% and 20.0%, respectively. The primary contributions lie in the dataset, executable environment, and evaluation design.
Critical strengths
Strength 1: The data resource establishes a complete interactive training pipeline.
The dataset integrates tasks, low-level mouse/keyboard operations, intermediate screenshots, and execution verification, uniformly supporting supervised training, reinforcement learning, and closed-loop evaluation. Each reference solution is verified via actual replay, making it more suitable for multi-step interaction research than datasets providing only images and answers. The inclusion of 20 task categories and a unified strategy further enhances the resource's research utility.
Strength 2: The necessity of pixel-level scoring.
For fixed-model click predictions, replacing mask-based judgment rules with tight bounding boxes cleanly isolates the impact of scoring metrics. Additional experiments reveal substantial biases in localization evaluation, strongly justifying the adoption of mask-based designs.
Critical weaknesses
Weakness 1: The incremental value of core supervision is relatively entangled, mixing protocol adaptation with task capability learning.
Pass@1 increases from 11.4% to 70.5% from base model to SFT, while submission rate rises from 20.5% to 96.8%. The paper notes that the base model rarely submits in the 16 task categories requiring explicit submission. Thus, while this comparison proves the overall training pipeline is effective, it fails to disentangle how much gain stems from visual reasoning versus learning to submit and operate fixed interfaces. Reasoning annotations are generated by teachers post-hoc after observing correct actions; verifying their correctness does not prove these explanations are essential for learning.
Weakness 2: Current generalization cannot sufficiently distinguish transferable capabilities from adaptation to specific assets or interfaces.
Appendix C explicitly states that each task category uses a fixed layout. Appendix H reports that 276 out of 282 test source images for Image Matching appear in the training set. This is a significant concern. Although the paper includes external testing, external baselines are weak and affected by interface failures: I note that UI-TARS entered the actual task interface only 11 times out of 1,200 Arkose/FunCAPTCHA samples on Halligan. Such disparities conflate interface initialization capability with captcha-solving ability.
Questions for authors
Question 1: Under identical conditions (same 12K training instances, same backbone, and comparable training budget), how do internal and external Pass@1 and submission rates change when reasoning text is removed, retaining only screenshot-action supervision? If the base model's non-submission issue were first corrected via explicit submission prompts or minimal protocol examples, how much of the original performance gap would remain?
Question 2: After incorporating strong baselines capable of reliably entering the task interface into external evaluations, does CaptchaAgent's advantage still hold?
4 comments
Response to Reviewer kExQ (Part 2/2): W2, Q2
2. W2/Q2: Distinguishing transferable capability from adaptation to specific assets and interfaces
Weakness 2: Current generalization cannot sufficiently distinguish transferable capabilities from adaptation to specific assets or interfaces. Appendix C explicitly states that each task category uses a fixed layout. Appendix H reports that 276 out of 282 test source images for Image Matching appear in the training set. This is a significant concern. Although the paper includes external testing, external baselines are weak and affected by interface failures: I note that UI-TARS entered the actual task interface only 11 times out of 1,200 Arkose/FunCAPTCHA samples on Halligan. Such disparities conflate interface initialization capability with captcha-solving ability.
Question 2: After incorporating strong baselines capable of reliably entering the task interface into external evaluations, does CaptchaAgent's advantage still hold?
(a) Assets: source images shared between training and testing. Since 276 of 282 images in Image Matching appear in training, we ran three checks on source images shared between training and testing:
- On the 12 types with no shared images across 2,400 test puzzles, the SFT model scores 63.3, the RL model scores 64.4, and the untrained model scores 3.0, or 30.0 with coordinate adaptation and forced submission.
- We removed test puzzles that share images with training data and rescored the remaining puzzles. The macro-average changes by no more than 0.2 points across four overlap definitions (Table R3).
- Within Image Matching, the 107 puzzles whose combinations of reference images and answers appeared in the SFT data score 97.9, while the remaining 93 puzzles score 98.5. Models never trained on our data already get 90.0-95.0 on this type.
Table R3. Pass@1 before and after removing test puzzles that share images with the training data
| Removal criterion | Types / puzzles kept | Types excluded | SFT (before → after) | RL (before → after) |
|---|---|---|---|---|
| Any image in the puzzle appears in the full training set | 14 / 2,752 | Coordinates, Dart Count, Hold Button, Image Matching, Image Recognition, Rotation Match | 62.0 → 62.2 | 63.0 → 63.2 |
| Any image in the puzzle appears in the SFT subset | 15 / 2,894 | Dart Count, Hold Button, Image Matching, Image Recognition, Rotation Match | 64.3 → 64.2 | 65.4 → 65.4 |
| The answer-determining image appears in the full training set | 16 / 3,151 | Coordinates, Hold Button, Image Matching, Image Recognition | 66.1 → 66.3 | 67.2 → 67.4 |
| The answer-determining image appears in the SFT subset | 17 / 3,294 | Hold Button, Image Matching, Image Recognition | 67.9 → 67.9 | 69.1 → 69.1 |
Types with fewer than 50 remaining test puzzles are excluded. Before and after filtering, Pass@1 (%) is macro-averaged over the same retained types. The full training set contains 2,100 puzzles per type, while the SFT subset contains 600.
(b) Interface: additional external baselines (Q2). We agree that UI-TARS's results primarily reflect its failure to enter task interfaces. We added two additional baselines:
Table R4. External benchmark comparison with additional baselines (Pass@1).
| Benchmark | Full SFT 9B | CaptchaAgent (RL) | UI-TARS-1.5-7B | GUI-Owl-1.5-8B | Holo-3.1-9B |
|---|---|---|---|---|---|
| OWC | 47.2 | 51.0 | 14.0 | 20.2 | 25.55 |
| Halligan | 13.6 | 20.0 | 0.9 | 26.3 | 21.54 |
These results distinguish between the two types of transfer you mentioned:
- New task content. OWC463 primarily evaluates this, and our policy outperforms the three baselines in the table.
- New interfaces. On Halligan, GUI-Owl-1.5-8B and Holo-3.1-9B achieve Pass@1 scores of 26.3 and 21.54, respectively, compared with 20.0 for our policy. Interface initialization also remains a limitation: our training data contains no start screens, and our policy enters the puzzle in 84% of start-screen episodes.
Therefore, the advantage holds on new task content, whereas it fails to extend to new interfaces.
(c) Interface: fixed layout per category. We will state this limitation in the revised paper. Fixed layouts let CaptchaArena focus on solving puzzles, whereas transferring to unseen interfaces requires different layouts and landing pages. Thus, this is a direction for the next version of the dataset.
We hope the above response addresses your questions, and we welcome further discussion if you have any additional questions.
Sorry for my late reply. After carefully reading all the reviews and the authors’ rebuttal, I have decided to maintain my original score. Best of luck with your paper!
Thank you for taking the time to read our responses and the other reviews. And thansk for your positive assessment of our work. We will incorporate the discussion into the final version.
Response to Reviewer kExQ (Part 1/2): W1, Q1
Thank you for your thoughtful review and constructive feedback. We appreciate your positive assessment. Below, we address your concerns and questions.
1. W1/Q1: Distinguishing protocol adaptation from problem-solving capability
(a) How much of the original gap remains after fixing submission and action formats first. Following your suggestion, we tested untrained Qwen3.5-9B with forced submission at the end of the episode. In Table 3 of the paper, the untrained model and CaptchaAgent use the same pixel action interface. Thus, we also tested the untrained model with native coordinate adaptation by converting positions from Qwen's 0–1000 convention to screen pixels before execution. Table R1 reports the result. The SFT model scores 69.0 in this single-run evaluation, compared to the five-run mean of 70.5 in the paper:
Table R1. Untrained Qwen3.5-9B with forced submission, compared with the SFT model.
All models are evaluated with one attempt per test puzzle. Pass@1 and submission rates are macro-averaged over the 20 types. Submission accuracy is computed separately for self-submitted and forced-submitted episodes within each type, then averaged over types with at least one corresponding submission. Forced submission is applied at episode end only if the model has not submitted.
Therefore, protocol adaptation indeed explains part of the original gap, yet a substantial gap remains even after addressing both submission and action formats.
(b) Whether reasoning text is necessary. The teacher writes the reasoning text after observing verified actions. Its correctness does not imply that it is necessary for learning. We completed training under the conditions you provided: the same 12K puzzles, the same backbone, and the same training recipe and optimization steps. We only removed the <think> content so the model learns purely "screenshot → action". Table R2 reports its in-domain and external Pass@1 and submit rates. We will add per-type results to the revised paper.
Table R2. SFT with and without reasoning text.
Following the paper's evaluation protocol: five rollouts per puzzle on CaptchaArena and one on each external benchmark. All metrics are macro-averaged over task types.
The table R2 shows that results in the setting without reasoning are higher than the reasoning results in the original paper. Through case analysis, we found that most of the gap stems from inaccurate localization. The reasoning text mainly describes the decision process, while precise localization is supervised by the action targets. We will analyze how the two interact in the revised paper.