acceptodds
Under review as a conference paper at ICLR 2027

Training Needs Trustworthy Worlds: Verified Synthetic Web Environments for Agent Learning

Abstract

Synthetic web environments can scale agent training, but plausible pages may hide unreachable goals, contradictory records, or invalid state updates. These defects confound policy failure with environment failure. We study environment reliability as a source of supervision for compact web agents. Our framework represents a website as a scaffold of pages, navigation, backend records, tasks, and state-change markers. Bounded verification and local repair establish executable task traces; marker-constrained updates and backend-derived rewards connect visible actions to persistent state. Across 500 environments in six domains, task feasibility increases from 48.6% to 94.8%. With terminal rewards fixed, verification raises held-out PPO success from 19.4% to 42.6%; adding dense state rewards reaches 58.7%. Controls separate these effects from feasible-task filtering and reward granularity. The resulting policies also improve on WebArena-compatible tasks, WebShop, and MiniWoB++ through a shared DOM interface, without policy-side LLM calls. The evidence supports bounded, DOM-grounded interaction learning while exposing remaining limitations in noisy web dynamics and verification coverage.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
web agents, synthetic environments, environment verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_Xs7WOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies defects in LLM-generated synthetic web environments (unreachable goals, invalid state updates, inconsistent rewards) and argues that such defects corrupt the training signal. It proposes a pipeline that generates environments, checks them with deterministic checks and LLM verifiers, repairs the detected defects, and re-validates them. Compact policies (8.9M–42M parameters) are then trained with PPO on the repaired environments. On 500 generated sites, task feasibility rises from 48.6% to 94.8%, held-out PPO success rises from 19.4% to 42.6% (and to 58.7% with dense state rewards), and the policies are also evaluated on WebArena-compatible tasks, WebShop, and MiniWoB++.

The paper presents itself as a training paper. Its stated contribution is “a unified environment-level formulation of reliable training supervision and a controlled evaluation of its effect on agent learning” (direct quote from §1), and its headline results are policy-learning gains. That central result is that policies learn better in cleaner environments. This is an expected result and is quite well-established and adds little new insight (W1). It is also shown only for small non-LLM policies (8.9M–42M parameters), with no experiments on small or medium-sized LLMs (in the range of 1B-30B models), so it does not establish whether the trend holds for the models the community trains today (W2).

The genuinely interesting part of the paper is its premise of automated environment verification and repair. In the current paper, however, this is supporting machinery rather than the main object of study. The repair mechanism is not specified in detail and it is not compared against other environment-generation pipelines (W3). Making it the central focus would change the paper's experimental layout, results, baselines, and ablations substantially.

Recommendation: Reject. I encourage the authors to revisit the paper's central thesis, re-centering it on automated environment verification and repair and to resubmit with the experiments that such a thesis requires.

Critical strengths
  • Interesting and novel premise. Automatically verifying and repairing generated environments, with task feasibility backed by replayable traces rather than an LLM's judgement (§4.1, Eq. 5), is a worthwhile direction. It is, however, not the paper's central thesis (see W3).
  • Careful reliability analysis of the verify/repair pipeline. The labeled audit of 80 environments and 1,286 defects (Table 10) reports precision, recall, F1, repair success, and false-repair rate. The full method has the best recall (87.2), F1 (89.3), and repair success (90.8%) among the construction baselines; rule-based checks remain better on precision (96.1) and false repairs (1.2% vs. 2.7%). This is complemented by an executability audit with an independent analyzer and author-audited replay (Table 9) and a blinded human execution study (Table 8).
  • Insightful analysis of which defects hurt learning. Figures 5 & 6 and §5.4 re-inject each defect type and measure the drop in PPO success. Feasibility defects cause the largest drop (24.3 points), followed by marker (17.3), structural (14.4), consistency (10.3), and semantic (4.7) defects. The analysis also shows that how often a defect occurs does not predict how much damage it does: semantic defects are more common but less harmful, while feasibility and marker defects are rarer but more damaging. This is useful guidance on which defects a repair pipeline should fix first, and it supports the verification-and-repair direction (W3).
  • Well-controlled training experiments. The training comparisons are task-count matched, separate verification from reward shaping (Table 2), use three seeds, and report confidence intervals. The execution is solid; the issue is that the question they answer has an expected answer (W1).
Critical weaknesses
  • The central training result is expected and does not establish anything new.

    • The paper's thesis is a training claim. The stated contribution (§1), the abstract, and the headline results (Fig. 4, Table 2) all argue that training on verified environments improves policy learning.
    • The direction is expected. The raw environments have only 48.6% feasible tasks and an average of 12.4 defects per site (Table 1). That a policy learns better once impossible tasks and broken state updates are removed is the expected outcome, not a new finding.
    • The finer-grained results refine, rather than change, this conclusion. Repair helps beyond simply filtering out infeasible tasks (36.4 vs. 58.7 with dense reward, Table 2). These measure the size of an expected effect; they do not reveal a new mechanism or overturn an existing belief.
  • No results on LLMs, so the trend is not established for the models the community actually trains.

    • Only small, non-LLM policies. All trained policies are action scorers of 8.9M, 15M, and 42M parameters. LLMs appear only as zero-shot baselines. No small or medium-sized LLM (e.g. 1B–8B) is fine-tuned or RL-trained, so there is no evidence of how the gains scale.
    • It's not obvious that PPO training gains will be seen for larger, pretrained models. A pretrained LLM already knows much about web pages and tasks, so it may be far less affected by environment defects than a small scorer without pretraining. Without such results, the training findings are of limited interest to the community in the AI current landscape.
  • The most interesting contribution, automated verification and repair, is under-developed; making it central would change the paper substantially.

    • The repair mechanism is not specified. §4.2 states what repairs achieve (“Repairs restore links or controls, reconcile values with canonical records, and repair marker schemas”) and Algorithm 1 says only “Propose a local edit.” The AI-use statement says LLMs “support … repair,” but the model, prompt, or algorithm that produces an edit is never given — unlike the verifier prompts in A.7.
    • The verification and repair loop is never run on an independent environment-generation pipeline. It only works on the authors' own scaffold format. Their pipeline needs state-change markers, typed records with declared bindings, a site specification, and invariants. However, real sites and existing benchmarks expose none of these.
    • Not evaluated as a contribution in its own right. The baselines are verification methods (rule-based, single-LLM, self-consistency, AutoGen); there is no comparison with other environment-generation pipelines. Only headline numbers appear in the main text (§5.2, §5.6); the full audit tables (Tables 8–11) are in the appendix.
    • Re-centering the paper on this contribution would require new baselines, cross-pipeline experiments, and ablations of the repair step. That's a different paper, hence the recommendation to resubmit.
Questions for authors
  • Which model produces repair edits, with what prompt or algorithm, and how is each edit kept local?
  • Can the verify→repair loop (in it's current state) be applied to environments from an independent generation pipeline and how does it compare with other environment-generation pipelines?
  • Will the environments and code be released? How many site specifications were written, how were they authored, and can you include one specification and one complete generated environment?
Reviewer_rLZtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a framework that automatically creates web environment to scale up web agent training. This framework automatically detect errors in the synthesized environment and tasks and it repair by itself. Using this framework, authors created 500 environments. They also showed the effectiveness of the generated environment with PPO experiment, which results in a improvement in the success rate from 19.4% to 42.6%.

Critical strengths

[S1] The ablation study has been done regarding to various of aspects. Also, by removing each component make notable score drops

Critical weaknesses

[W1] Lack Novelty. I guess there is some work related to synthesis of web environment [1,2]. They did not make the difference between these related work and their work clearly. The authors need to clearly explain what’s the novelty beyond the related work and relation to these work.

[W2] The experimental setups don’t make sound. Why did the authors use very small policy model, such as bidirectional GRU, not LLM? Also, these policy models are not built to generate action sequences, but they are trained to score action candidates and pick the best action. When actions are predicted in this way, input action (e.g., typing a search query for a search box) cannot be covered in this experiment.

[1] AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machines

https://www.semanticscholar.org/paper/AutoWebWorld%3A-Synthesizing-Infinite-Verifiable-Web-Wu-Peng/6099ff1fe728a3172b9dea63b5222537bc8927cb Questions

[2] Safe and Scalable Web Agent Learning via Recreated Websites

https://www.semanticscholar.org/paper/Safe-and-Scalable-Web-Agent-Learning-via-Recreated-Chae-Park/87f39233f656faa1c00cefded82ea2044508e650#related-papers

Questions for authors

[Q1] How can we actually know the task executability metric is correct? Can you do a human study to check whether the proposed metric works as intended?

Reviewer_kxHTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reliability of synthetic web environments for agent training, by proposing a framework that generates structured website scaffolds, verifies task feasibility and state consistency, and applies local repairs before policy training. Persistent state updates are handled through validated event markers, while backend task predicates provide intermediate rewards for PPO. Experiments on 500 synthetic environments show substantial improvements in task feasibility and policy success. The trained compact policies also achieve gains on DOM-compatible subsets of WebArena, WebShop, and MiniWoB++.

Critical strengths

The paper addresses an important issue in synthetic agent training, that is invalid environments can produce misleading supervision. The evaluation includes reward controls, feasible-task filtering, component ablations, and independent execution audits. In particular, the improvement from 19.4% to 42.6% under terminal-only rewards supports the importance of environment verification beyond reward shaping. Also, the marker-based simulator reduces LLM calls from 17.8 to 2.3 per episode while maintaining relatively high state fidelity. The external evaluations also suggest that the improved training environments can benefit compact policies without policy-side LLM calls.

Critical weaknesses
  1. There are several relevant published works, including AutoWebWorld, VeriEnv and InfiniteWeb that have already explored executable or verifiable synthetic web environments and their use in agent learning. In particular, InfiniteWeb also employs task-centric verification and verifiable evaluators for RL training. The paper does not adequately position its contributions against these approaches or provide a direct comparison. This makes it difficult to assess the novelty and significance of the proposed framework beyond integrating existing techniques.

  2. The replay certificate establishes the existence of a successful trajectory for a particular initial state and recorded transition realization, rather than general environment correctness. The framework also relies on generated task specifications and invariants, which may themselves be incomplete. Although author acknowledge these limitations, the experiments do not fully establish whether repaired environments preserve the intended task semantics across varying initial states and state-update realizations. The reported modifications to task constraints further motivate this concern.

Questions for authors
  1. How does the proposed framework differ technically from AutoWebWorld, VeriEnv, and InfiniteWeb? Which components represent the main methodological contribution beyond existing environment verification and programmatic reward frameworks? A controlled comparison or a more precise distinction would substantially strengthen the novelty claim.

  2. Appendix C.12 reports that 2.4% of task constraints are modified during repair. Can the authors clarify which constraints are changed and demonstrate that these edits preserve the original task objectives? In particular, how does the method perform when task completion predicates are held strictly fixed?

  3. How stable are the feasibility certificates across different initial states and repeated realizations of the LLM-based state writer? Reporting repeated execution success rates would help establish whether the certified environments provide reliable supervision beyond a single successful replay.

Reviewer_JqGKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for constructing synthetic web environments for agent training. It combines rule-based checks, LLM-based verification, local repair, and constrained state updates to improve task executability and training feedback. Experiments on 500 synthetic websites report improved environment quality and policy performance, with additional transfer results on a WebArena-compatible subset, WebShop, and MiniWoB++.

Critical strengths
  1. The paper addresses a practical problem: defects in synthetic environments can produce misleading training feedback.

  2. The repair pipeline considers dependencies, rechecks affected components, and rejects unsuccessful edits.

  3. The experiments include reward controls, component ablations, human execution studies, and external benchmark evaluation.

Critical weaknesses
  1. Repair quality is not independently established. Passing more checks or completing more tasks does not necessarily mean the environment is more correct. How are repairs checked for preserving task requirements, and who provides the reference labels?

  2. The downstream baselines are limited. Most policy comparisons use raw or rule-checked environments. The paper should also train policies on environments repaired by its stronger construction baselines, such as AutoGen, to establish the practical advantage of the proposed framework.

  3. Some defects may be avoidable with better generation and tooling. Would explicitly providing dependencies and constraints in the generation prompt prevent these errors? Would a coding agent that runs tests and repairs failures achieve similar results? Neither alternative is adequately tested.

  4. Generalization of the repair pipeline is unclear. How much manual specification is needed for a new domain? Transfer of the trained policy does not show that the verification and repair system itself generalizes.

  5. “Trustworthy” does not precisely describe what is evaluated. The experiments assess executability and internal consistency. These are useful properties, but “environment reliability” would more accurately describe the contribution.

Questions for authors

see weakness

Reviewer_Ty85OpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a scalable framework for automatically generating and validating synthetic web environments. It addresses the problem of faulty agent training in a defective environment that includes broken navigation links, invalid state updates, and contradictory records. The proposed framework generates a structured scaffold of web tasks, verifies and locally repairs it, and trains a compact agent on the reliable synthetic environment. Across 500 synthetic environments spanning six domains, the framework increases bounded task feasibility from 48.6% to 94.8%. Training compact agents on the verified environments also improves held-out task success from 19.4% to 42.6%. The resulting policies also improve tasks of three other popular benchmarks.

Critical strengths
  1. The paper addresses an important issue of web agent training on a large scale reliable web environment.
  2. The paper does not rely on LLMs, rather employ deterministic structural checks to validate the tasks. It tests each task by actually replaying the steps to see if they reach the goal
  3. The paper presents extensives experiments conducted in their own synthetic environment and also tested the trained agents on three outside benchmarks to validate the effectiveness
Critical weaknesses

– The Method section is very hard to follow. An explicit example and a diagram can help to understand the main objective and process of each of three subsections. –The checks only confirm that the environment matches with its own generated specification. They do not confirm that the specification is correct. – Web agent training methods that the paper cites, such as WebRL and Agent Q, are not compared with the proposed solution. – The paper does not report formal statistical significance tests. Several key tables (Table 1 and Table 2) only provide point estimates.

Questions for authors

Please address the weak points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.