acceptodds
Under review as a conference paper at ICLR 2027

CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs

11 views

Abstract

Interactive CAPTCHAs remain challenging for computer-use agents, while existing datasets face trade-offs among type coverage, interaction fidelity, and trajectory supervision. To address these gaps, we present CaptchaArena, the first large-scale, fine-grained training dataset for interactive CAPTCHA solving. It contains 50K puzzles across 20 CAPTCHA types and 5 interaction modes, with every solution verified through execution. CaptchaArena provides 50K screenshot-action trajectories, including 46K with step-by-step reasoning annotations. It also includes fine-grained pixel-mask annotations for irregular targets. Using CaptchaArena, we train CaptchaAgent, a single 9B policy for all 20 CAPTCHA types, with supervised fine-tuning followed by reinforcement learning. The environment verifier directly provides the RL reward. Supervised fine-tuning reaches 70.5 Pass@1, and reinforcement learning further improves it to 71.7, while also improving performance on two external benchmarks. These results demonstrate the value of large-scale, fine-grained computer-use supervision for training interactive CAPTCHA agents.

Primary area:
datasets and benchmarks
Keywords:
training dataset, computer-use agents, interactive CAPTCHAs, GUI agents, vision-language models, reinforcement learning, GUI grounding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kExQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CaptchaArena, a dataset comprising 50K instances covering 20 captcha categories and 5 interaction modes. It provides execution-verified screenshot-action trajectories, with 46K entries including reasoning annotations and pixel-level masks for irregular click regions. Using Qwen3.5-9B as the backbone, the authors perform SFT with 12K training trajectories followed by GRPO based on environmental feedback to obtain a unified model, CaptchaAgent. Pass@1 accuracy improves from 11.4% (base model) to 70.5% (SFT), reaching 71.7% after RL. On external benchmarks Open CaptchaWorld and Halligan, it achieves 51.0% and 20.0%, respectively. The primary contributions lie in the dataset, executable environment, and evaluation design.

Critical strengths

Strength 1: The data resource establishes a complete interactive training pipeline.

The dataset integrates tasks, low-level mouse/keyboard operations, intermediate screenshots, and execution verification, uniformly supporting supervised training, reinforcement learning, and closed-loop evaluation. Each reference solution is verified via actual replay, making it more suitable for multi-step interaction research than datasets providing only images and answers. The inclusion of 20 task categories and a unified strategy further enhances the resource's research utility.

Strength 2: The necessity of pixel-level scoring.

For fixed-model click predictions, replacing mask-based judgment rules with tight bounding boxes cleanly isolates the impact of scoring metrics. Additional experiments reveal substantial biases in localization evaluation, strongly justifying the adoption of mask-based designs.

Critical weaknesses

Weakness 1: The incremental value of core supervision is relatively entangled, mixing protocol adaptation with task capability learning.

Pass@1 increases from 11.4% to 70.5% from base model to SFT, while submission rate rises from 20.5% to 96.8%. The paper notes that the base model rarely submits in the 16 task categories requiring explicit submission. Thus, while this comparison proves the overall training pipeline is effective, it fails to disentangle how much gain stems from visual reasoning versus learning to submit and operate fixed interfaces. Reasoning annotations are generated by teachers post-hoc after observing correct actions; verifying their correctness does not prove these explanations are essential for learning.

Weakness 2: Current generalization cannot sufficiently distinguish transferable capabilities from adaptation to specific assets or interfaces.

Appendix C explicitly states that each task category uses a fixed layout. Appendix H reports that 276 out of 282 test source images for Image Matching appear in the training set. This is a significant concern. Although the paper includes external testing, external baselines are weak and affected by interface failures: I note that UI-TARS entered the actual task interface only 11 times out of 1,200 Arkose/FunCAPTCHA samples on Halligan. Such disparities conflate interface initialization capability with captcha-solving ability.

Questions for authors

Question 1: Under identical conditions (same 12K training instances, same backbone, and comparable training budget), how do internal and external Pass@1 and submission rates change when reasoning text is removed, retaining only screenshot-action supervision? If the base model's non-submission issue were first corrected via explicit submission prompts or minimal protocol examples, how much of the original performance gap would remain?

Question 2: After incorporating strong baselines capable of reliably entering the task interface into external evaluations, does CaptchaAgent's advantage still hold?

4 comments
AuthorsOpenReview ↗

Response to Reviewer kExQ (Part 1/2): W1, Q1

Thank you for your thoughtful review and constructive feedback. We appreciate your positive assessment. Below, we address your concerns and questions.

1. W1/Q1: Distinguishing protocol adaptation from problem-solving capability

Weakness 1: The incremental value of core supervision is relatively entangled, mixing protocol adaptation with task capability learning. Pass@1 increases from 11.4% to 70.5% from base model to SFT, while submission rate rises from 20.5% to 96.8%. The paper notes that the base model rarely submits in the 16 task categories requiring explicit submission. Thus, while this comparison proves the overall training pipeline is effective, it fails to disentangle how much gain stems from visual reasoning versus learning to submit and operate fixed interfaces. Reasoning annotations are generated by teachers post-hoc after observing correct actions; verifying their correctness does not prove these explanations are essential for learning.

Question 1: Under identical conditions (same 12K training instances, same backbone, and comparable training budget), how do internal and external Pass@1 and submission rates change when reasoning text is removed, retaining only screenshot-action supervision? If the base model's non-submission issue were first corrected via explicit submission prompts or minimal protocol examples, how much of the original performance gap would remain?

(a) How much of the original gap remains after fixing submission and action formats first. Following your suggestion, we tested untrained Qwen3.5-9B with forced submission at the end of the episode. In Table 3 of the paper, the untrained model and CaptchaAgent use the same pixel action interface. Thus, we also tested the untrained model with native coordinate adaptation by converting positions from Qwen's 0–1000 convention to screen pixels before execution. Table R1 reports the result. The SFT model scores 69.0 in this single-run evaluation, compared to the five-run mean of 70.5 in the paper:

Table R1. Untrained Qwen3.5-9B with forced submission, compared with the SFT model.

SettingPass@1Self-submit rateForced-submit shareSubmission accuracy: self / forced
Qwen3.5-9B (unified pixel interface) + forced submission18.420.3%79.3%37.4% / 8.9%
Qwen3.5-9B (with coordinate adaptation) + forced submission34.876.8%22.8%38.4% / 16.7%
Full SFT (with reasoning)69.093.7%—72.0% / —

All models are evaluated with one attempt per test puzzle. Pass@1 and submission rates are macro-averaged over the 20 types. Submission accuracy is computed separately for self-submitted and forced-submitted episodes within each type, then averaged over types with at least one corresponding submission. Forced submission is applied at episode end only if the model has not submitted.

  1. With forced submission, the untrained model increases from 11.4 to 18.4. After we also adapt the action format, it reaches 34.8.
  2. A gap of 34.2 points from the SFT model remains. Also, the untrained model's accuracy after submission is 38.4%, compared to 72.0% for the SFT model.
  3. On the four types where a click automatically submits without needing submission actions, the untrained model gets 56.6 in Table 3, or 57.9 with coordinate adaptation, while the SFT model gets 81.8.

Therefore, protocol adaptation indeed explains part of the original gap, yet a substantial gap remains even after addressing both submission and action formats.

(b) Whether reasoning text is necessary. The teacher writes the reasoning text after observing verified actions. Its correctness does not imply that it is necessary for learning. We completed training under the conditions you provided: the same 12K puzzles, the same backbone, and the same training recipe and optimization steps. We only removed the <think> content so the model learns purely "screenshot → action". Table R2 reports its in-domain and external Pass@1 and submit rates. We will add per-type results to the revised paper.

Table R2. SFT with and without reasoning text.

ModelCaptchaArena Pass@1Submit rateOWC463 Pass@1Submit rateHalligan Pass@1Submit rate
With reasoning (the paper)70.596.8%47.290.6%13.653.4%
Without reasoning75.199.9%51.998.0%21.377.6%

Following the paper's evaluation protocol: five rollouts per puzzle on CaptchaArena and one on each external benchmark. All metrics are macro-averaged over task types.

The table R2 shows that results in the setting without reasoning are higher than the reasoning results in the original paper. Through case analysis, we found that most of the gap stems from inaccurate localization. The reasoning text mainly describes the decision process, while precise localization is supervised by the action targets. We will analyze how the two interact in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer kExQ (Part 2/2): W2, Q2

2. W2/Q2: Distinguishing transferable capability from adaptation to specific assets and interfaces

Weakness 2: Current generalization cannot sufficiently distinguish transferable capabilities from adaptation to specific assets or interfaces. Appendix C explicitly states that each task category uses a fixed layout. Appendix H reports that 276 out of 282 test source images for Image Matching appear in the training set. This is a significant concern. Although the paper includes external testing, external baselines are weak and affected by interface failures: I note that UI-TARS entered the actual task interface only 11 times out of 1,200 Arkose/FunCAPTCHA samples on Halligan. Such disparities conflate interface initialization capability with captcha-solving ability.

Question 2: After incorporating strong baselines capable of reliably entering the task interface into external evaluations, does CaptchaAgent's advantage still hold?

(a) Assets: source images shared between training and testing. Since 276 of 282 images in Image Matching appear in training, we ran three checks on source images shared between training and testing:

  1. On the 12 types with no shared images across 2,400 test puzzles, the SFT model scores 63.3, the RL model scores 64.4, and the untrained model scores 3.0, or 30.0 with coordinate adaptation and forced submission.
  2. We removed test puzzles that share images with training data and rescored the remaining puzzles. The macro-average changes by no more than 0.2 points across four overlap definitions (Table R3).
  3. Within Image Matching, the 107 puzzles whose combinations of reference images and answers appeared in the SFT data score 97.9, while the remaining 93 puzzles score 98.5. Models never trained on our data already get 90.0-95.0 on this type.

Table R3. Pass@1 before and after removing test puzzles that share images with the training data

Removal criterionTypes / puzzles keptTypes excludedSFT (before → after)RL (before → after)
Any image in the puzzle appears in the full training set14 / 2,752Coordinates, Dart Count, Hold Button, Image Matching, Image Recognition, Rotation Match62.0 → 62.263.0 → 63.2
Any image in the puzzle appears in the SFT subset15 / 2,894Dart Count, Hold Button, Image Matching, Image Recognition, Rotation Match64.3 → 64.265.4 → 65.4
The answer-determining image appears in the full training set16 / 3,151Coordinates, Hold Button, Image Matching, Image Recognition66.1 → 66.367.2 → 67.4
The answer-determining image appears in the SFT subset17 / 3,294Hold Button, Image Matching, Image Recognition67.9 → 67.969.1 → 69.1

Types with fewer than 50 remaining test puzzles are excluded. Before and after filtering, Pass@1 (%) is macro-averaged over the same retained types. The full training set contains 2,100 puzzles per type, while the SFT subset contains 600.

(b) Interface: additional external baselines (Q2). We agree that UI-TARS's results primarily reflect its failure to enter task interfaces. We added two additional baselines:

Table R4. External benchmark comparison with additional baselines (Pass@1).

BenchmarkFull SFT 9BCaptchaAgent (RL)UI-TARS-1.5-7BGUI-Owl-1.5-8BHolo-3.1-9B
OWC47.251.014.020.225.55
Halligan13.620.00.926.321.54

These results distinguish between the two types of transfer you mentioned:

  1. New task content. OWC463 primarily evaluates this, and our policy outperforms the three baselines in the table.
  2. New interfaces. On Halligan, GUI-Owl-1.5-8B and Holo-3.1-9B achieve Pass@1 scores of 26.3 and 21.54, respectively, compared with 20.0 for our policy. Interface initialization also remains a limitation: our training data contains no start screens, and our policy enters the puzzle in 84% of start-screen episodes.

Therefore, the advantage holds on new task content, whereas it fails to extend to new interfaces.

(c) Interface: fixed layout per category. We will state this limitation in the revised paper. Fixed layouts let CaptchaArena focus on solving puzzles, whereas transferring to unseen interfaces requires different layouts and landing pages. Thus, this is a direction for the next version of the dataset.

We hope the above response addresses your questions, and we welcome further discussion if you have any additional questions.

Sorry for my late reply. After carefully reading all the reviews and the authors’ rebuttal, I have decided to maintain my original score. Best of luck with your paper!

Thank you for taking the time to read our responses and the other reviews. And thansk for your positive assessment of our work. We will incorporate the discussion into the final version.

Reviewer_7GjSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CaptchaArena, a large-scale CAPTCHA dataset for training computer-use agents to solve interactive CAPTCHAs. This helps the computer-use agents solve more practical scenarios. The authors collect 50K screenshot action trajectories and also fine-tune a Qwen 3.5 9B model, showing improved performance on the constructed validation benchmark.

Critical strengths
  1. This paper introduces a dataset that can collect materials and construct solver trajectories on its own, which is efficient when scaling

  2. The proposed benchmark covers a wide range of CAPTCHA types, ranging from easy to hard

Critical weaknesses
  1. The evaluation protocol conflates "learning to submit" with "learning to solve." As noted in the paper (Sec. 5.2), a significant portion of the improvement stems simply from the model learning to submit. However, since a large portion of GUI agents already learn this submission as general behavior from the general GUI training, the authors should isolate the performance gains attributable to the dataset itself

    A suggested baseline is to use the Qwen model, either prompted or integrated into a harness that forces a submission in the final step while also reporting the accuracy rate for submitted episodes to separate the effect

  2. There is also an overlap of materials between the train set and the test set. For example, in image matting, 276 out of the 288 images are overlapped, and in these sectors, the scores are all in the range of 96 to 99

The authors should provide a cleaner division or report the results after removing those sectors

  1. There is a 15-point gap between the pass@1 and pass@5 metrics following SFT, whereas RL only narrows this by 0.4 points. This discrepancy may also lack enough analysis and explanation, such as reward design, a frozen vision encoder, and a max 15 steps
Questions for authors
  1. What is the exact human verification process for CoT labeling? What are the exact pass rate and fail rate of the filter, and the scale of the human fix?

  2. As far as I know, the OpenCAPTCHA World contains categories that are intentionally constructed to be hard and not very realistic for actual website deployment. This paper follows the types from the OpenCAPTCHA World. In the paper, when transferring to a more realistic benchmark set, Halligan, the success ratio is also only 20%. Can the authors further report which parts of the constructed dataset have a stronger connection to realistic websites, and how they improve performance on Halligan (e.g., being removed from the training set causes a drop in Halligan)? This breakdown could serve as a mixing ratio for subsequent research to train GUI agents for more realistic purposes

If all weaknesses and questions are addressed, I will consider raising my score.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 7GjS (Part 1/6): W1

1. W1: Separating “learning to submit” from “learning to solve”

Weakness 1: The evaluation protocol conflates "learning to submit" with "learning to solve." As noted in the paper (Sec. 5.2), a significant portion of the improvement stems simply from the model learning to submit. However, since a large portion of GUI agents already learn this submission as general behavior from the general GUI training, the authors should isolate the performance gains attributable to the dataset itself

A suggested baseline is to use the Qwen model, either prompted or integrated into a harness that forces a submission in the final step while also reporting the accuracy rate for submitted episodes to separate the effect

We agree that submission behavior and solving ability should be examined separately.

Table 3 compares the same model before and after training. Specifically, untrained Qwen3.5-9B and CaptchaAgent share the same interface, prompts, and tools, and they differ only in weights. In addition, the evaluator executes the model's output coordinates directly as screen pixels. In these conditions, the untrained model gets a Pass@1 of 11.4. Following your suggestion, we added two controls. The first one forces a submission: if a model does not submit before one episode ends, the evaluator submits for it. The second one adds coordinate adaptation on top of forced submission. Because Qwen outputs relative coordinates in the 0–1000 range by default, we convert them into screen pixels before execution. Thus, we can observe what the untrained model can achieve with this interface support.

Coordinate adaptation converts the locations output by the model under its 0–1000 coordinate convention into screen pixels before execution.

This additional evaluation was conducted on the same test set and according to the same grading rules as in Table 3. Each model went through evaluation once for each puzzle, and results were macro-averaged over the 20 types of puzzles. The table below reports Pass@1, submit rate, and accuracy after submission.

Table R1. Untrained Qwen3.5-9B with forced submission, compared with the SFT model.

SettingPass@1Self-submit rateForced-submit shareSubmission accuracy: self / forced
Qwen3.5-9B (unified pixel interface) + forced submission18.420.3%79.3%37.4% / 8.9%
Qwen3.5-9B (with coordinate adaptation) + forced submission34.876.8%22.8%38.4% / 16.7%
Full SFT (with reasoning)69.093.7%—72.0% / —

We calculate accuracy after submission as the ratio of right submissions to all submitted episodes. We compute it for each type individually and then average it over the types that have submissions. The SFT model gets a Pass@1 of 69.0 in a single run, compared with 70.5 averaged over 5 runs in the paper.

  • With forced submission alone, the untrained model increases its Pass@1 from 11.4 to 18.4.
  • With coordinate adaptation, the untrained model submits on its own in 76.8% of the episodes and reaches 34.8 with forced submission.
  • Even so, it remains behind the SFT model at 69.0 by 34.2 points. Also, it gets a lower accuracy after submission of 38.4% compared to 72.0%.
  • On the four types that submit automatically after the solving action and need no submit action, the untrained model scores 56.6 in Table 3, or 57.9 with coordinate adaptation, compared to the SFT model's score of 81.8.

Fixing the coordinate format and submission improves the untrained model's score, but it remains far behind the SFT model even with both fixed. Combined with the gain on the automatically submitting types, these results confirm that CaptchaArena improves solving accuracy as well as submission behavior. We will add these two controls in the revised paper and report submit rate and accuracy after submission separately.

This decomposition also holds when moving from the SFT model to the RL model. The RL model increases the Pass@1 score by 1.23 points, where 0.34 comes from a higher submit rate and 0.89 comes from higher accuracy after submission.

AuthorsOpenReview ↗

Response to Reviewer 7GjS (Part 2/6): W2

2. W2: Train–test overlap

Weakness 2: There is also an overlap of materials between the train set and the test set. For example, in image matting, 276 out of the 288 images are overlapped, and in these sectors, the scores are all in the range of 96 to 99

The authors should provide a cleaner division or report the results after removing those sectors

Thank you for bringing this up. In Appendix H, we note that some types contain images that appear in both the training and test data. In Image Matching, for instance, 276 of 282 images found in testing also appear in the training data. In this type, each puzzle picks its own reference image and distractors and has its own layout, so the same image does not mean the same puzzle. To test how much this reuse affects performance, we ran three checks.

(1) Removing the types with overlaps. When comparing training and testing images by content, 12 of the 20 types share no images at all. Following your suggestion, we keep only these 12 types, for a total of 2,400 test puzzles. When we recompute the scores from the per-type results in Table 3, untrained Qwen3.5-9B, the SFT model, and the RL model score 3.0, 63.3, and 64.4, respectively. Also, in the extra test, the Qwen baseline with coordinate adaptation and forced submission scored 30.0. Across these 12 types, training still brings a clear gain.

(2) Removing only the overlapping puzzles. Check 1 eliminates all types with overlaps. Here we keep every type and remove only the puzzles that share images with the training set. We then rescore the remaining puzzles and compare the scores with those of the same types before removing the overlapping puzzles.

We define “overlapping images” in four different ways, one for each row of the table below:

  • Which image we check: any image used within the puzzle, or only the image that decides the answer, such as the reference image or the correct option.
  • Which training puzzles we compare against: all 2,100 training puzzles of that type, or only the 600 used for SFT. If this filtering leaves a type with fewer than 50 of its 200 test puzzles, we exclude that type. The table lists these types.

Table R2. Performance before and after removing overlapping test puzzles.

Removal criterionTypes / puzzles keptTypes excludedSFT (before → after)RL (before → after)
Any image in the puzzle appears in the full training set14 / 2,752Coordinates, Dart Count, Hold Button, Image Matching, Image Recognition, Rotation Match62.0 → 62.263.0 → 63.2
Any image in the puzzle appears in the SFT subset15 / 2,894Dart Count, Hold Button, Image Matching, Image Recognition, Rotation Match64.3 → 64.265.4 → 65.4
The answer-determining image appears in the full training set16 / 3,151Coordinates, Hold Button, Image Matching, Image Recognition66.1 → 66.367.2 → 67.4
The answer-determining image appears in the SFT subset17 / 3,294Hold Button, Image Matching, Image Recognition67.9 → 67.969.1 → 69.1

In all four rows, the change is within ±0.2 points. Thus, even after we remove puzzles that share images with the training set, the results stay stable for the types we keep. For these types, removing the overlapping puzzles does not cause a systematic drop in the macro-averaged score.

(3) Within-type comparison. In Image Matching, 107 test puzzles have a reference image and answer pair that appears unchanged in the SFT data. The SFT model reaches 97.9 Pass@1 on these puzzles and 98.5 on the other 93 puzzles. Across all 9 types where this split is possible, no type shows clearly better performance on puzzles whose reference image and answer pair appears in the SFT data. We will add these results in the revised paper and clearly list the types we keep.

AuthorsOpenReview ↗

Response to Reviewer 7GjS (Part 3/6): W3

3. W3: The Pass@1–Pass@5 gap after SFT

Weakness 3: There is a 15-point gap between the pass@1 and pass@5 metrics following SFT, whereas RL only narrows this by 0.4 points. This discrepancy may also lack enough analysis and explanation, such as reward design, a frozen vision encoder, and a max 15 steps

We appreciate your bringing up this topic. In Table 5, RL improves Pass@1 by 1.2 and Pass@5 by 0.8. The difference between the two measures has thus decreased by only 0.4, as you pointed out. After SFT, 51.7% of test puzzles are solved in all five rollouts, 14.0% fail in all five, and 34.3% are mixed. RL raises the all-correct share to 54.3%, an increase of +2.6 points, and lowers the mixed share to 32.5%, a decrease of −1.8 points. The largest all-correct increases, from 7.5 to 11.5 points, appear in Connect Icon, Geometry Click, Coordinates, and Rotation Match, while the changes in the three types with the largest gaps are not statistically significant. To explain why the gap exists, we analyze the factors you noted and the underlying task properties:

(1) The three factors you observed.

  • Reward design: A verified solution receives nearly 1 reward versus at most 0.35 otherwise. However, total signal was limited. We trained the model for only 29 updates, which totaled 3,712 trajectories across the 20 types. This included fewer than 20 puzzles from each type with the largest gaps. Also, 21.6% of groups had identical rewards across all rollouts and thus provided no learning signal.
  • Frozen vision encoder: On the three types with the largest gaps, the SFT model submits in at least 99.8% of rollouts, so nearly every failure there is a submitted wrong answer. Updating only the language model likely constrains gains on puzzles that require fine visual precision. We have not yet run RL with an unfrozen vision encoder.
  • 15-step limit: Only 3.0% of SFT rollouts reach this limit, and this share drops to 2.1% after RL. The three types with the largest gaps usually finish within 1–3 steps, and at most 0.1% of their rollouts reach the limit. Thus, this limit does not explain their gaps. Reaching the limit is concentrated in Dice Count, at 41% for SFT and 24% for RL, where the policy often exhausts the turn budget.

(2) Task properties: all-or-nothing grading and action precision. CAPTCHA puzzles use all-or-nothing grading and require precise actions. A single imprecise action, such as an inaccurate click or drag position, invalidates the whole episode. These puzzles are short. The three types with the largest gaps are Slide Puzzle, Bingo, and Pick Area, where each gap exceeds 30 points. They usually finish within 1–3 steps. In these three types, 72–88% of puzzles are mixed. With five rollouts, the model gets more chances to produce one fully correct episode. This is the main source of the Pass@1–Pass@5 gap.

Reviewer_4zBjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduced two artifacts:

  • CaptchaArena: a dataset containing 50K CAPTCHA puzzles across 20 types. The CAPTCHA images are generated by prompting ChatGPT or procedurally using Blender. Each type has 2100 training, 200 validation, and 200 test CAPTCHAs. The ground truth actions come from three sources: generation-time labels, manual annotation, and renderer-derived masks. To enable SFT, The puzzles are annotated step by step by a frontier model to produce a reasoning that goes along with the action (with a judge to verify if the annotation matches the answer). To enable RL and interactive evaluation, the puzzles are structured as an executable browser task with the answer being a browser program, admitted only after its replay is accepted by the site’s own verifier.

  • CaptchaAgent: a Qwen-3.5-9B policy trained in two phases (SFT, then RL) on all 20 types of CaptchaArena (600 from each type for SFT, and 1500 for GRPO RL), the 200 validation are used for monitoring, while the 200 testing are used for final evaluation. CaptchaAgent shows a significant uplift and reaches 70.5 Pass@1 after SFT and 71.7 after RL compared with 11.4 for the untrained backbone.

The results show significant uplift and the remaining gap to humans is concentrated in a small number of types.

Critical strengths
  • The data generation and annotation pipeline is novel, scalable and easy to understand and replicate.
  • The results show significant uplift and a 9B model is able to exceed gemini and other frontier models with the training pipeline described in the paper
  • The paper is very easy to read and well-structured, the method is mostly well-described
Critical weaknesses
  • The RL benefit is very small compared to the effort and amount of data, no explanation was provided.
  • Some puzzle types actually worsened by the RL process (Misleading Click, Click Order), no explanation was provided.
  • The data sampling choices/ablations are weakly justified: It's not clear what the strategy was in choosing the subset of data for SFT vs. RL, why not train the agent on the same 600 tasks in RL again? Failure modes in RL would be different to SFT anyway, so overfitting is less of a concern.
Questions for authors
  • Do you have a hypothesis why RL is not giving larger gains, and in some cases, making results worse for some types?
  • I would be interested to see if a larger model would have more capacity to learn more from the RL, have you considered running the same experiment with Qwen3.5-27B model?
  • Please clarify your method in choosing which puzzles were used in SFT vs. RL, why not reuse the puzzles included in SFT in RL as well?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4zBj (Part 1/2): W1, Q1, W2, Q2

Thank you for reviewing the paper carefully and for providing constructive comments and your positive assessment of our work. Below we address each weakness together with the related question, including a new paired analysis of the RL stage.

1. Why the overall RL gain is modest

Weakness 1: The RL benefit is very small compared to the effort and amount of data, no explanation was provided.

Question 1: Do you have a hypothesis why RL is not giving larger gains, and in some cases, making results worse for some types?

We present our RL as a simple, small-scale baseline. Sampling in a real browser takes about 9 hours per update step of 8 puzzles × 16 rollouts, or 128 trajectories. Thus, compute constraints prevented us from training on the full RL pool of 2,643 puzzles in the released setting. We reached the best checkpoint reported in the paper after 29 update steps. These steps covered 3,712 trajectories across 232 puzzles.

Even this small run gives a real and expected improvement. We ran both models 5 times on every test puzzle in a paired per-puzzle evaluation. This evaluation shows an average Pass@1 gain of +1.23 points, and 16 of 20 types improve. Also, we see external gains on Open CaptchaWorld from 47.2 to 51.0 and on Halligan from 13.6 to 20.0. Part of the gain occurs at the format level through more reliable submission. For example, the Dice Count submit rate rises from 54.4% to 76.6%. Overall, 0.34 points of the gain come from a higher submit rate and 0.89 points come from higher accuracy after submission. Two factors account for the modest gain:

  1. Limited training coverage. Because of the compute cost of real browser sampling, we trained on only 232 puzzles out of the pool of 2,643 puzzles.
  2. Limited effective learning signal. GRPO treats all 16 attempts at the same puzzle as one group. In 21.6% of groups, all attempts received the same reward, which provides no learning signal.

In addition, we froze the vision encoder, and the language model parameters changed only slightly. The relative L2 norm change between the SFT model and the RL model reached only 2.9×10⁻⁴. Future work includes large-scale RL on the full pool or exploring other algorithms such as PPO. We release the dataset and its executable environment to support this future work.

2. Types that got worse after RL

Weakness 2: Some puzzle types actually worsened by the RL process (Misleading Click, Click Order), no explanation was provided.

  1. Click Order (−3.1). The average Pass@1 dropped by 3.1 points. The submit rate held steady from 96.9% to 97.3%. Also, two kinds of errors increased: first, glyph reading mistakes and incorrect clicking order, and second, mistakes in click positioning. Because we used only 12 Click Order puzzles in training, this drop may come from interference from training on other puzzle types, which our current analysis cannot yet confirm.
  2. Misleading Click (−0.2). Average performance changed by only −0.2 points. Out of 200 tested puzzles, 8 improved and 9 dropped. Thus, performance remained stable.

We will add these analyses to the revised paper.

3. Scaling RL to Qwen3.5-27B

Question 2: I would be interested to see if a larger model would have more capacity to learn more from the RL, have you considered running the same experiment with Qwen3.5-27B model?

We agree that running SFT and RL on Qwen3.5-27B would show how model scale affects RL gains. Running this experiment requires completing SFT on the 27B model before launching GRPO. For the 9B model, each GRPO update contains 8 puzzles × 16 rollouts = 128 trajectories sampled in a real browser and takes about 9 hours on average. Scaling to the 27B model requires much more compute and time. Thus, we cannot finish it before the rebuttal deadline.

During the discussion period we completed the first half of this experiment on a reduced budget. We fine-tuned Qwen3.5-27B on 10% of the SFT data, which took 17 hours on 8 A100 GPUs. For comparison, the 9B model was fine-tuned on the same 10% subset with the same number of epochs and effective batch size, which took 17 hours on 4 RTX A6000 GPUs. We report Pass@1 on a fixed 400-puzzle subset of the CaptchaArena test split (20 puzzles per type, one attempt each).

Table R1. Comparison of 9B and 27B models trained on 10% of the SFT data.

ModelSFT dataCaptchaArenaSubmit rate
9B10%17.5043.00%
27B10%72.5099.75%

Running GRPO on top of the 27B model is beyond what we can finish before the deadline. We leave it as future work.

AuthorsOpenReview ↗

Response to Reviewer 4zBj (Part 2/2): W3, Q3

4. W3/Q3: How puzzles were split between SFT and RL

Weakness 3: The data sampling choices/ablations are weakly justified: It's not clear what the strategy was in choosing the subset of data for SFT vs. RL, why not train the agent on the same 600 tasks in RL again? Failure modes in RL would be different to SFT anyway, so overfitting is less of a concern.

Question 3: Please clarify your method in choosing which puzzles were used in SFT vs. RL, why not reuse the puzzles included in SFT in RL as well?

Thank you for your observation regarding our work.

  1. How the split was made. Out of the 2,100 training puzzles of every type, we used 600 puzzles for SFT and kept the rest for RL. Then we ran the SFT model 5 times on each candidate puzzle and picked puzzles individually based on the results. We preferred puzzles with mixed results across the five attempts and made sure we used all types of puzzles. As a result, we got an RL training pool of 2,643 puzzles. Thus, the 1,500 puzzles per type formed the candidate pool for RL, and we used a subset of them in RL training.
  2. Why SFT puzzles were not reused. We avoided puzzles that SFT had already seen and memorized. The model likely solves such puzzles in all rollouts. Thus, GRPO gets no reward difference within a group and no learning signal.
  3. Reusing SFT puzzles is also reasonable. We agree with the point you made: when the model solves puzzles online, it makes mistakes absent from the previous demonstrations. Therefore, as long as the results are mixed, GRPO can still learn when we reuse SFT puzzles. Thus, reusing SFT puzzles is also a reasonable choice. Excluding SFT puzzles is a design choice of this experiment. We will explain this design consideration in the revised paper.

Thank you again for your constructive feedback. We hope these clarifications address your concerns, and we would be happy to provide further details if needed.

Dear Reviewer 4zBj,

Thank you again for your review and positive assessment. We have posted our responses above, along with a general summary. We are happy to clarify anything further before the discussion period ends.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We thank the Area Chair and Reviewers 4zBj, 7GjS, and kExQ for their constructive feedback. We summarize the main points below.

  • Submission vs. solving (7GjS, kExQ). With forced submission and native coordinates, untrained Qwen3.5-9B reaches 34.8 Pass@1. The SFT model reaches 69.0. We will re-evaluate all baselines under native coordinate conventions.
  • Train–test overlap (7GjS, kExQ). Removing overlapping test puzzles changes macro Pass@1 by at most 0.2 points.
  • Reasoning annotations (kExQ). SFT without reasoning text scores 75.1, above 70.5 with reasoning. We will report both settings.
  • RL (4zBj, 7GjS). RL ran for only 29 updates and adds 1.2 points. We will present it as a small-scale baseline.
  • Model scale (4zBj). With 10% of the SFT data, Qwen3.5-27B reaches 72.5 Pass@1, compared with 17.5 for the 9B model. RL on 27B is left as future work.
  • External transfer (7GjS, kExQ). In a 10%-data ablation, removing five types with Halligan counterparts lowers Pass@1 on those counterparts from 11.8 to 2.6. CaptchaAgent leads on Open CaptchaWorld (51.0). On Halligan it scores 20.0, below GUI-Owl-1.5-8B (26.3) and Holo-3.1-9B (21.54). We will state this limitation.
  • Ethics (7GjS). We will expand the misuse discussion and use gated release.

We agree with Reviewer 7GjS that the main value of this work is the environment and pipeline for scalable, verified training data. The final version will reflect this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.