acceptodds
Under review as a conference paper at ICLR 2027

Weblica: Scalable and Reproducible Training Environments for Visual Web Agents

Abstract

The web is complex, open-ended, and constantly changing, making it challenging to scale training environments for visual web agents. Existing data collection efforts remain limited to offline trajectories for supervised fine-tuning or a handful of simulated environments for reinforcement learning (RL), failing to capture the diversity of the web. Training directly on live websites could alleviate this, but it is slow, brittle, irreproducible, and can have unintended side effects on real websites. We propose Weblica (Web Replica), a framework for constructing reproducible and scalable web environments. At its core, Weblica uses coding agents to synthesize interactive websites grounded in real-world websites and core web navigation skills, allowing us to scale RL training to thousands of diverse environments served locally. To assess how well synthetic websites can substitute for real ones, we further build a proxy for live-web training by caching real websites at the HTTP level, capturing and replaying stable visual states while preserving interactive behavior. Both synthetic and cached real environments yield substantial improvements over the base model, with purely synthetic RL achieving the best overall performance. Our best model, Weblica-8B, outperforms open-weight baselines of similar size across multiple web navigation benchmarks while using fewer inference steps, scales favorably with additional test-time compute, and is competitive with API models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
web agents, GUI agents, multimodal agents, reinforcement learning, synthetic data, RL environments

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q5m8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces Weblica, a framework for generating scalable and reproducible web environments for training visual web agents. By constructing synthetic interactive websites, Weblica enables large-scale reinforcement learning without relying on live websites. Experiments demonstrate improved performance across multiple web navigation benchmarks.

Critical strengths

S1. Scalability

Weblica provides a scalable approach to generating diverse interactive web environments. The framework could be useful to the community if made publicy available.

Critical weaknesses

W1. Limited Scientific Novelty

While Weblica demonstrates scalability, its scientific novelty relative to prior work, such as VeriEnv, InfiniteWeb, and AutoWebWorld, appears limited. The contributions primarily lie in scaling up environment generation and RL training rather than introducing fundamentally new methodological insights. As a result, the work appears to be primarily an engineering contribution, with limited scientific novelty beyond existing approaches.

W2. Insufficient Ablation of Capability-Driven Synthesis

The paper lacks a critical ablation study to validate its primary methodological distinction: capability-driven synthesis. In particular, it does not compare environment generation with and without capability conditioning under matched environment counts and RL training budgets. Consequently, it remains unclear whether capability-driven synthesis itself contributes to the reported performance gains or whether these gains are primarily driven by the scale of the generated training data.

Questions for authors
  1. What are the key scientific contributions of Weblica beyond scaling, particularly compared to VeriEnv, InfiniteWeb, and AutoWebWorld?
  2. Can the authors provide a controlled ablation to demonstrate the effectiveness of capability-driven synthesis?
  3. How does the reliability of the LLM-based judge compare to programmatic verification, and how does this choice affect RL training?
5 comments
AuthorsOpenReview ↗

Response to Reviewer q5m8 (1/3) – Scientific contributions

Thank you for the constructive review and for recognizing that Weblica offers a scalable approach to generating diverse interactive web environments, which could be useful to the community. We address your concerns below.

Scientific contributions (W1, Q1)

We respectfully disagree that the contribution is mainly engineering. Weblica differs from the works you mention both conceptually and in focus. Conceptually, two of the three do not train with online RL, and all three train at a much smaller scale. AutoWebWorld uses offline single-step RL on 29 websites, and VeriEnv uses rejection-sampling fine-tuning on 97 websites and leaves RL to future work. InfiniteWeb does use online GRPO, but on 600 tasks and with rewards limited to what its generated code checks. In contrast, Weblica trains online multi-turn RL on thousands of environments and tasks. Importantly, Weblica rewards each task with an LLM judge that reads natural-language criteria and the screenshots of the episode, so no task needs a hand-written checker, and Weblica can reward a much broader set of open-ended tasks, including free-text answers (see Q3).

Weblica also achieves strong results. In a system-level comparison on Online-Mind2Web, the only benchmark all four works report, Weblica-8B reaches 39.2 pass@1 on all 300 tasks, 10.6 points above its base model. The other three report 28.7 (InfiniteWeb), 24.6 (VeriEnv) and 14.0 (AutoWebWorld). Weblica-8B also improves substantially on the other two live-web benchmarks, DeepShop and WebTailBench (Table 1).

Please also note that our focus goes beyond environment synthesis. Besides synthesizing websites, we build a second reproducible alternative to live-web training, HTTP-level caches of real websites, and compare the two under the same recipe (Sec. 5.4). Both improve substantially over the base model, and each comes with its own trade-off. Caching keeps real content but supports only the recorded interactions and the tasks the recording agent could solve. Synthesis can target the interactions agents need, and it performs better overall (see the table in our answer to W2 below). We believe this comparison of environment choices is useful to the community beyond our specific pipeline, and Reviewer HPbX also found it "worth studying".

In summary, Weblica makes three scientific contributions beyond scaling.

  • Synthesized websites can substitute for real ones as RL environments. Under the same recipe and number of tasks, RL on synthesized websites outperforms RL on HTTP-level caches of real websites on average across three live-web benchmarks.
  • Online RL with an LLM-judge reward works at scale. Because the judge needs no per-task checker, we run online multi-turn RL on thousands of tasks and environments. The resulting policies improve substantially over their base models on three live-web benchmarks, at three different model sizes.
  • Diverse synthesized environments enable offline development. Conditioning each site on a website category, a visual style and an interaction capability yields thousands of distinct sites. We found that the held-out synthesized sites track live-web performance on Online-Mind2Web closely (App. D.4), so models can be developed and selected without access to live websites.
AuthorsOpenReview ↗

Response to Reviewer q5m8 (2/3) – Capability-driven synthesis and LLM judge

Capability-driven synthesis vs. scale (W2, Q2)

We would like to point out that capability conditioning is what allows synthesis to scale in the first place. It is a common observation that generative models produce highly similar outputs when prompted repeatedly with the same request [1, 2]. As a result, the quantity of synthetic data scales easily, but its diversity does not [3]. The common remedy is to condition each request on attributes sampled in advance [1, 4]. This yields more diverse data and better downstream models than a plain prompt [1], and sampling the attributes from a taxonomy of topics or capabilities further increases diversity and coverage [2, 5]. We observed the same when generating websites, as repeated requests for the same capability converged to similar domains and styles (App. B.3). Category and style seeds address this for content and appearance. The capability seed, drawn from an inventory extracted from agent trajectories including failed ones (App. B.2), decides which interactions a site exercises, so we obtain thousands of sites that exercise different interactions instead of variants of the same site. For us, scale and capability conditioning are therefore intertwined.

Regarding scale, please also note that simply scaling up is not enough. To demonstrate this, we cache real websites from the InSTA (v3) task pool and train on the same number of tasks (10K) with the same recipe. Synthesized environments still perform better overall (Sec. 5.4).

pass@1, 8B, same recipe and 10K RL tasksOM2WDeepShopWTBAvg.
Base (Qwen3-VL-8B)28.624.121.824.8
RL on cached real websites35.335.830.233.8
RL on synthesized websites (Weblica-8B)39.234.233.535.6

LLM judge vs. programmatic verification (Q3)

Programmatic verification needs a checker for each task, written into or alongside the site's code, and it works only when success shows up in a stored state or an exact answer string. Prior work restricts its tasks accordingly. VeriEnv admits only "tasks with a single, well-defined answer", InfiniteWeb checks flags it inserts into localStorage, and AutoWebWorld does not score the content of answers.

An LLM judge, in contrast, scores any task from its natural-language criteria and the screenshots of the episode. It needs no per-task code and also accepts free-text answers, e.g., a Weblica-val task asks what an error code describes and which equipment it affects (App. E.2, Fig. 11). This is why we use it as the RL reward (Sec. 4). As reported in App. F.5, our judge agrees with human labels on 88% of 100 training tasks and reaches 91.0% accuracy on 300 Online-Mind2Web trajectories, with open-weight judges on par. Its remaining errors act as reward noise, yet the trained policy improves its base model by 10.8 points on average on live-web benchmarks scored by their own official judges.

Please also note that LLM judges are an established choice for web agents. Live-web benchmarks such as Online-Mind2Web, DeepShop and WebTailBench score agents with LLM judges rather than programmatic checks, given the diversity of real-web tasks, and LLM judges are increasingly used as rewards for training web and GUI agents, e.g., in DigiRL [6] and PAE [7]. All our live-web results are measured with the benchmarks' official judges.

[1] Yu et al., Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias, NeurIPS 2023.

[2] Davidson et al., Reasoning-Driven Synthetic Data Generation and Evaluation, TMLR 2026.

[3] Ge et al., Scaling Synthetic Data Creation with 1,000,000,000 Personas, arXiv 2024.

[4] Samvelyan et al., Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts, NeurIPS 2024.

[5] Li et al., Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models, arXiv 2024.

[6] Bai et al., DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning, NeurIPS 2024.

[7] Zhou et al., Proposer-Agent-Evaluator (PAE): Autonomous Skill Discovery for Foundation Model Internet Agents, ICML 2025.

AuthorsOpenReview ↗

Response to Reviewer q5m8 (3/3) – Public availability

Public availability (S1)

The paper gives full details for reproducing the pipeline, as summarized in our Reproducibility Statement. Appendix B covers capability extraction, the seed pools, the coding-agent harness and its tools, task and criteria generation, all prompts, per-site statistics and generation costs. Appendix C details how the cache is constructed. Section 4 and Appendix F describe the training recipe, hyperparameters, compute, action space and judge prompt. The environments are static HTML/CSS/JS, and open-weight judges perform on par with GPT-4o (App. F.5), so training on them does not require a proprietary model. Generation is a one-time step that other capable coding agents could also perform, and the rest of the pipeline builds on open-weight models (Qwen3-VL), open-source tools (Playwright and veRL) and a public task dataset (InSTA (v3)).

We hope these answers clarify our contributions, and we are glad to discuss any remaining questions.

Reviewer_HPbXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Weblica uses coding agents to generate local websites and tasks for training visual web agents. It also builds cached versions of real websites for comparison. The authors generate 2,810 synthetic sites, apply SFT followed by RL with an LLM judge, and evaluate on three live-web benchmarks. The 8B model's mean pass@1 improves from 24.8% to 35.6%.

Critical strengths

The scale is useful: the paper supplies thousands of local environments and reports transfer across three live-web benchmarks and several model sizes. The synthetic-versus-cache comparison is worth studying, and the construction costs and judge audits help assess whether the pipeline can be reused.

Critical weaknesses

The headline efficiency comparison is also not convincing. Table 1 compares Weblica's pass@2 or higher with published results under different retry rules. Counting a task as solved when any attempt succeeds is different from delivering one successful outcome, unless success can be identified. At pass@1, Weblica's live-web mean is 35.6%, compared with MolmoWeb's reported 42.4%. Also, the capability inventory was extracted from all 300 Online-Mind2Web tasks. This is not evidence of copied answers, but that benchmark is not independent of the synthesis design; the other two benchmarks provide cleaner transfer evidence.

Questions for authors
  1. Can the efficiency comparison use the same attempt budget and failure-handling rules for both methods? Otherwise, please separate pass@1 results from pass@k scaling.

  2. Was Online-Mind2Web used for checkpoint or hyperparameter selection as well as capability extraction? How does performance change with a capability inventory drawn from an independent source?

1 comment
AuthorsOpenReview ↗

Response to Reviewer HPbX – Matched comparison with MolmoWeb-8B and capability source

Thank you for the careful and constructive review. Your suggestions improved our paper, and we address both questions below.

Matched attempt budgets (Q1)

"Can the efficiency comparison use the same attempt budget and failure-handling rules for both methods? Otherwise, please separate pass@1 results from pass@k scaling."

We will report pass@1 separately from pass@k in the paper. We also evaluated Weblica-8B and MolmoWeb-8B under the same attempt budgets and failure-handling rules, at 30 and 100 steps, as follows:

To make this an apples-to-apples comparison, we run MolmoWeb-8B using its released code, prompt, action space and observation history, which is the current screenshot plus the text of its last 10 steps. We run both MolmoWeb-8B and Weblica-8B in the same standard local Playwright browser, and each benchmark's official judge scores both. Following MolmoWeb-8B's protocol, we re-run an episode up to 10 times when the environment fails. An episode that fails on a bot-check page also gets up to 3 fresh browser sessions. Both models get the same retries.

As Weblica-8B was trained and evaluated with 30-step episodes, we explore two potential ways to run it at 100 steps.

  • Matched history. Weblica-8B runs one 100-step episode with the same observation history as MolmoWeb-8B (the current screenshot plus the text of the last 10 steps).
  • Agent-initiated retries. Weblica-8B stays within the 30-step horizon it was trained for. Whenever it reports failure (no answer at the step limit, an empty answer, or an environment failure), it starts a fresh 30-step attempt, up to 3 attempts. Please note that Weblica-8B itself decides when to start a new attempt, without an oracle, an external judge or ground truth. Each task still gets one delivered answer, and a task uses at most 90 steps.
pass@1 (%)OM2WDeepShopWTBAvg.
30 steps
MolmoWeb-8B35.722.021.726.4
Weblica-8B37.534.736.636.3
100 steps
MolmoWeb-8B39.328.026.831.4
Weblica-8B, matched history41.239.033.337.8
Weblica-8B, agent-initiated retries (≤3 × 30 steps)39.838.741.740.1

Weblica-8B does significantly better at both budgets. Averaged over the three benchmarks, it leads MolmoWeb-8B by 9.8 points at 30 steps. At 100 steps, it leads by 6.5 points with matched history and by 8.7 points with agent-initiated retries. Even at 30 steps, Weblica-8B outperforms MolmoWeb-8B at 100 steps on average, which suggests that it uses its steps more efficiently.

Thank you again. These results made our comparison much clearer, and we will add them, together with the evaluation protocol, to the paper.

Use of Online-Mind2Web and an independent capability source (Q2)

"Was Online-Mind2Web used for checkpoint or hyperparameter selection as well as capability extraction? How does performance change with a capability inventory drawn from an independent source?"

Online-Mind2Web was not used for checkpoint or hyperparameter selection. We select both on Weblica-val, which consists of held-out synthesized sites. We found Weblica-val to be a reliable proxy for selection, as it tracks live-web performance on Online-Mind2Web closely (App. D.4).

To test whether the capabilities depend on Online-Mind2Web, we repeated capability extraction (App. B.2) on trajectories from 2,675 InSTA (v3) websites, only 7 of which share a domain with Online-Mind2Web. As a control, we also ran it on an equally sized Online-Mind2Web subset, using Claude Sonnet 5.5 for both.

InSTA (v3)OM2W (control)
Extracted capabilities that fit our 144 groups90.9%93.3%
Our 2,810 sites whose group appears in the source94.9%97.7%

Capabilities from the broad web fit our 144 groups almost as well as a fresh Online-Mind2Web sample, and the groups they cover account for nearly all of our sites. The groups not observed on InSTA (v3) are rare in both sources, e.g., file upload. This suggests that an inventory drawn from an independent source would yield nearly the same capability groups.

Indeed, as you observed, the other two benchmarks show transfer. The gains there are similar to or larger than those on Online-Mind2Web, both in absolute terms (+10.1 and +11.7 vs. +10.6 pass@1 over the base) and in relative terms (+42% and +54% vs. +37%).

We will add this analysis to the paper.

We hope these results address both of your questions, and we are glad to provide further details.

Reviewer_dAVMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Weblica, an automated framework to construct reproducible and scalable web environments to solve the environment scarcity issue in web agent training.

Weblica uses agent coding tools to write interactive websites grounded in real-world websites and common web navigation skills. By using this method, the authors scaled RL training to thousands of diverse environments (compared to under 10 in prior works) and achieved better performance on various benchmark datasets.

Critical strengths

This paper addresses an important issue: web RL environments are scarce and rely heavily on manual construction and annotation.

Critical weaknesses

The most critical weakness is the lack of reliable validation for the generated intent and criteria. Because it is also LLM-generated, it relies heavily on the performance of the task generation LLM. Also, for the LLM-as-a-judge agent, the lack of a verifiable reward may introduce noise into the training process.

  1. The criteria or the final state may not be reachable at all.
  2. There is no guarantee that the intent and the criteria actually correspond with each other.

A minor issue is that some other work on automatic environment and RL training data synthesis should also be discussed. for example: https://aclanthology.org/2026.acl-long.2037.pdf https://arxiv.org/abs/2605.17558

Questions for authors

Do you have any validation results? Like, probably a very small human annotation result on the generated task and criteria fidelity?

2 comments
AuthorsOpenReview ↗

Response to Reviewer dAVM – Validation of tasks and criteria

Thank you for the positive assessment and for highlighting the scarcity of web RL environments, which today rely heavily on manual construction. We address your concerns below.

Validation of tasks and criteria (W, Q)

"Do you have any validation results? Like, probably a very small human annotation result on the generated task and criteria fidelity?"

Following your suggestion, we manually reviewed 100 random training tasks, one per site. For each, we checked whether the task is feasible on its website, whether its criteria match its intent, and whether its intent is clear, i.e., understandable without knowing the site, with one reasonable reading and a clear deliverable. We started from a coding agent's attempt at each task, with screenshots, and then reviewed every task manually.

Feasible on its websiteCriteria match the intentIntent clearTasks passing
✓––95/100
–✓–97/100
––✓99/100
✓✓✓91/100

On reachability (1), 95 of the 100 tasks are feasible. The other 5 have site bugs that block or hinder the task, such as a checkout button with no handler. In RL, such tasks mostly produce groups in which every rollout fails, and, as noted in Sec. 4, our filter discards these groups.

On correspondence (2), intent and criteria match in 97 tasks. This is expected, since the coding agent writes each task together with its site and knows every ground-truth value (App. B.5). In the remaining 3, the criteria are too loose, e.g., they do not check that a form was submitted or accept any non-zero count, so they could reward an incomplete episode or a wrong answer. We also observe that no task rewards a goal other than the one its intent states.

Beyond the tasks themselves, please also note that we report the judge's alignment with human labels in App. F.5, with 88% agreement on 100 training tasks and 91.0% accuracy on 300 Online-Mind2Web trajectories. Reviewer HPbX also noted that these judge audits help assess whether the pipeline can be reused. Furthermore, while some noise remains in tasks and rewards, Weblica still transfers well to live-web benchmarks scored with their own judges, improving significantly over the base model at all three model sizes (2B, 4B and 8B).

Thank you for suggesting this study. We will add it to the paper, together with qualitative examples.

Related work

Thank you, we will add both works to our related work. Both synthesize verifiable data for text-based tool-calling agents. Trajectory2Task turns executed tool-call trajectories into tasks, and Firefly back-chains single-answer tasks from explorations of real APIs and trains with GRPO in a replay simulator, close in spirit to our cached environments. Weblica instead synthesizes the interactive environments themselves, for visual web agents, and writes tasks together with each environment.

We hope this addresses your concerns, and we are glad to answer any further questions.

Thank you for the response. I appreciate the effort put into the human verification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response – Summary of reviewer feedback and new analyses

We thank all reviewers for their thoughtful and constructive feedback as the discussion period comes to a close. We are encouraged that reviewers found that Weblica addresses an important problem, the scarcity of web RL environments, which today rely heavily on manual construction and annotation (dAVM). Reviewers noted that Weblica scales RL training to thousands of diverse environments, compared to fewer than ten in prior work (dAVM), and offers a scalable approach to generating diverse interactive web environments that could be useful to the community (q5m8). They highlighted that the resulting agents transfer across three live-web benchmarks and several model sizes (HPbX), found the comparison of synthesized and cached real websites worth studying, and considered the reported construction costs and judge audits helpful for assessing whether the pipeline can be reused (HPbX). In the discussion, reviewers acknowledged our findings on synthesized versus cached environments, online RL with LLM-judge rewards and offline development as valuable empirical findings and practical contributions (q5m8), and appreciated the effort put into the human verification of our tasks and criteria (dAVM). All three reviewers found the paper clear and easy to follow, and two reported no substantive concerns about its soundness (dAVM, HPbX).

Following the reviewers' suggestions, we added four analyses during the discussion. Please see the individual replies for details.

  • Human validation of tasks and criteria (dAVM). We manually reviewed 100 random training tasks, one per site. 95 are feasible on their website, 97 have criteria that match their intent, and 99 have a clear intent. The few infeasible tasks come from site bugs and mostly yield groups in which every rollout fails, which our RL filter discards.
  • Apples-to-apples comparison with MolmoWeb-8B (HPbX). We ran MolmoWeb-8B with its released code, prompt and observation history, using the same browser, step budgets and retry rules as for Weblica-8B. Weblica-8B does significantly better at both step budgets.
  • Capabilities from an independent source (HPbX). We repeated capability extraction on trajectories from InSTA (v3), a pool of websites from the broad web. 90.9% of the extracted capabilities fit our 144 capability groups, and for 94.9% of our sites, the group the site targets also appears in these trajectories. An inventory drawn from an independent source would thus yield nearly the same groups.
  • Controlled ablation of the capability seed (q5m8). We regenerated 100 random training sites without their capability seed, keeping everything else fixed. With the capability seed, the sites cover significantly more capability groups and spread their tasks more evenly across them. Category and style seeds play a complementary role and keep the content diverse, as without any seeds, generation collapses onto a few website categories.

We will add all four analyses to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.