acceptodds
Under review as a conference paper at ICLR 2027

Echoverse: Deep, Evolving Environments For Training Computer-Use Agents at Scale

11 views

Abstract

Training computer-use agents requires software application environments that the agents can interact with. Sandbox environments offer a better alternative to real ones, since they can be deterministic and safer to take actions in. However, creating these environments with sufficient depth and breadth of functionalities for training agents is non-trivial. We present Echoverse, a semi-automated pipeline that builds synthetic environments from specifications and target scenarios, with task outcomes verified against references derived from the environments' database. Echoverse works by co-evolving the environment and the model together in a loop. Specifically, for each step, we perform rollouts in our environments and use verifiers to discern between model successes/failures and environment issues. Based on the verifier analysis, we improve the environment and use the rollouts for training the model. By adjusting the environment with feedback from the rollouts, we are able to add functionalities needed to enable more tasks and increase the fidelity to real applications. We use Echoverse to produce a set of 10 environments covering different software use-cases. We show that fine-tuning on rollouts from environments verified only by unit tests and simple checks leads to poor transfer to real environments compared to fine-tuning on our environments (e.g., vs accuracy). % We show that fine-tuning on rollouts from "shallow” environments without further iterations leads to poor transfer to real environments compared to fine-tuning on "deep” environments from later iterations (e.g., vs accuracy). Building on these results, we introduce an RL training recipe that combines database grounded trajectory rewards with dense per-step feedback to address sparse supervision over long action sequences. We show that this recipe improves performance significantly in our synthetic environments (e.g., to pass@4).

Primary area:
reinforcement learning
Keywords:
RL, Computer Use, Agents, Environments, Self-Evolve, Post-training, Rewards, Verification, Synthetic, Data, Tasks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Bh6JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Echoverse, a framework for constructing and iteratively improving synthetic environments for computer-use agents. It jointly maintains environments, tasks, and verifiers, using agent rollouts to identify and repair defects. The authors build ten domain-specific environments and two capability-focused environments. Through SFT and RL experiments, they show that improved environment quality and targeted capability training can benefit agent performance. They also introduce an RL recipe combining database-grounded outcome rewards with step-level feedback, demonstrating substantial improvements in synthetic environments but limited gains on real-world benchmarks.

Critical strengths
  1. The paper addresses an important problem: agent failures may originate from environment defects or incorrect task verification rather than model limitations. Its co-evolution framework provides a practical approach to identifying and repairing these issues.
  2. Echoverse uses agent rollouts to identify and repair defects across environments, tasks, and verifiers, addressing an important source of unreliable training signals.
Critical weaknesses
  1. Co-evolution is a central contribution of this paper, however, its impact on downstream agent learning is mainly demonstrated through a single environment (EchoStay). The paper provides limited evidence that similar improvements consistently hold across the other constructed environments.
  2. Although verification is database-grounded, the final judgments still rely on LLMs. The framework lacks independent evaluation of verification accuracy, and its permissive handling of additional database modifications may introduce incorrect rewards.
Questions for authors
  1. Have the authors independently evaluated the correctness of generated task references and verifier judgments, particularly false-positive rewards and unintended database modifications?
  2. Is co-evolution itself responsible for the observed improvements, or are the gains primarily due to collecting more and better training trajectories?
  3. The SFT studies provide evidence of real-site transfer, but the synthetic-RL improvement on Online-Mind2Web is small and not statistically significant. What additional evidence supports the generalization benefits of the RL recipe, particularly beyond the observed improvements in termination and trajectory length?
1 comment

Thank you for the careful review, and for recognizing that environment and verifier defects are an important source of unreliable training signal. We appreciate the questions on co-evolution and verification and address each one below.

Co-evolution's impact on downstream agent learning is mainly demonstrated through a single environment (EchoStay).

Co-evolution is supported by three controlled comparisons, spanning SFT and RL and covering eight domains:

  1. EchoStay SFT: accuracy more than doubles, from 16.2% to 38.5%, with both checkpoints evaluated on the same repaired world.
  2. SFT depth study (Sec. 5): Allrecipes and Hugging Face environments built without co-evolution give no gain, or a loss, on live WebVoyager (80.0 → 75.0 and 48.0 → 48.0). Our environments give 85.0 and 65.0.
  3. RL across all six training environments (Fig. 6): a single co-evolution round on task definitions and references raises the mean@4 gain at 20 updates from 15.34 to 20.86 pp. Initialization, tasks, and rollout budget are held fixed.

Each controlled ablation requires rebuilding environments, recollecting trajectories, and retraining, so we could not run one per world. We observed the same pattern across all worlds during development, and Appendix D.6 documents rollout-driven repairs beyond EchoStay. One EchoForum task set goes from 0 to 36 of 37 tasks solved, and EchoChat's gradable tasks go from 34% to 99%. Without these repairs, the affected tasks would have produced wrong training signal.

Have the authors independently evaluated the correctness of generated task references and verifier judgments?

Yes, at two levels:

  • Automated: every task is checked by frontier agents with access to code, database, and browser.
  • Manual: QA is performed on sampled tasks.

A world ships only when 95% of claims and 95% of tasks pass, with no hard blockers, and in these checks errors were rare.

Extra database writes are allowed deliberately. Many applications log updates even when the agent only browses, so rejecting all extra writes would penalize correct behavior. When co-evolution surfaces false positives from a class of unintended changes, we add explicit constraints to the reference to disallow that class. The verifier itself improves through the same loop.

Is co-evolution itself responsible for the observed improvements, or are the gains due to more and better trajectories?

Data volume is controlled in every ablation, so the gains are not from more data:

  • SFT: task and trajectory counts differ by less than 10% between conditions. We will add the exact counts to the paper.
  • RL: both runs use the same tasks, initialization, and 192 rollouts per update, compared at exactly 20 updates on the same validation set.

The better trajectories are the mechanism, not a confound. A repaired environment makes more complex interactions feasible, so the same budget yields better data. This is exactly what co-evolution is designed to do.

What additional evidence supports the generalization benefits of the RL recipe?

The recipe generalizes to 290 held-out validation tasks that are disjoint from training:

  • mean@4 rises from 39.05% to 60.34%.
  • pass@4 rises from 64.83% to 77.59%.
  • Every one of the six environments improves.

The step-reward ablation isolates the recipe's contribution. Without step rewards, mean@4 at update 20 is 47.19%, versus 59.91% with them, a 12.73 pp gap under identical settings.

Online-Mind2Web is an out-of-distribution test by design. Our tasks are write-heavy workflows in login-gated applications, which live-web benchmarks cannot support, while OM2W is read-only research on public sites. RL also starts from an SFT checkpoint trained on 2,000 live-site tasks, which already captures most of the live-web gain. Despite both factors, RL still improves success (54.80% → 56.58%) and cuts non-termination by 4× (14.9% → 3.6%), with no loss in live-web accuracy.

Every concern raised is answered by controlled results already in the submission:

  • Co-evolution is validated in three controlled comparisons across SFT and RL.
  • Data volume is matched in every ablation.
  • The verifier is checked through automated and manual QA and repaired through the same loop.
  • The RL recipe generalizes to 290 held-out tasks, with every environment improving.

The only addition needed is the exact SFT counts. We do not believe a weak reject reflects this evidence, and we ask the reviewer to raise their score.

Reviewer_4NhAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Echoverse is an automated pipeline for constructing and repairing synthetic environments for CUA agents. It jointly develops tasks and verifiers using feedbacks. The results show large gains within synthetic environments and on held-out test sets.

Critical strengths
  1. Jointly checking applications, tasks, and verifiers addresses a practical bottleneck in environment construction.
  2. Held-out set experiments demonstrate transfer beyond the training distribution.
Critical weaknesses
  1. The verifier relies on an LLM and allows some additional database changes. For the database changes, it seems that outcome-based verification could be enough, and there is no need for an LLM-as-judge. Authors should check its judgments against human assessments, since these judgments affect both training and evaluation.
  2. The benefit of synthetic RL envs remains unclear. Task successes increase only from 154 to 159. These results do not yet establish a reliable improvement.
  3. It should be at least a small eval set that is based on real-world websites to see how synthesized website gains transfer into real-world scenarios.
Questions for authors

See weakness.

1 comment

Thank you for the review, and for highlighting both the joint checking of applications, tasks, and verifiers and the held-out transfer results. We take the presentation feedback seriously. Several results relevant to your concerns are spread across sections, so in the revision we will improve this, including a consolidated table of all real-website evaluations. We address each point below.

The verifier relies on an LLM and allows some additional database changes. Outcome-based verification could be enough, and there is no need for an LLM-as-judge.

Our verification is outcome-based. References come from SQL queries over the seeded database, and write tasks are graded on the sqldiff between the initial and final database state. The judge sees no screenshots or action history. It only compares the agent's answer or database diff against this grounded reference.

The LLM handles semantic matching that exact checks cannot:

  • Read answers vary in format, units, and rounding, e.g., $287.62 vs. $288, or 157286400 bytes vs. "150 MB".
  • Write diffs often contain incidental logging updates alongside the required change.

Exact-match verification would need hand-written checkers for thousands of tasks across ten environments, and would reject many correct outcomes.

For correctness, every task is checked by frontier agents with access to code, database, and browser, and manual QA is done on sampled tasks. A world ships only when 95% of claims and 95% of tasks pass. Verifier errors found in rollouts are repaired through co-evolution. For example, realigning EchoChat's verifier raised gradable tasks from 34% to 99%. When extra writes cause false positives, we add constraints to the reference that disallow that class of change.

The benefit of synthetic RL environments remains unclear. Task successes increase only from 154 to 159.

154 → 159 is an out-of-distribution check, not the main evidence. On 290 held-out validation tasks disjoint from training:

  • mean@4 rises from 39.05% to 60.34%.
  • pass@4 rises from 64.83% to 77.59%.
  • Every one of the six environments improves.

Removing step rewards drops mean@4 at update 20 from 59.91% to 47.19%, which isolates the recipe's contribution.

Online-Mind2Web is out of distribution by design. Our tasks are write-heavy workflows in login-gated applications, while OM2W is read-only research on public sites. RL also starts from an SFT checkpoint trained on 2,000 live-site tasks, which already captures most of the live-web gain. Despite this, RL still improves success and cuts non-termination by 4× (14.9% → 3.6%).

The value of synthetic environments is clearest against live-web RL with the same recipe (Sec. 6, App. F):

  • Each update takes 3.03× longer.
  • Each trajectory needs about 24× more judge requests.
  • Only 132–174 of 192 rollouts per update are usable after CAPTCHAs, rate limits, and browser failures.

Synthetic environments give exact resets and grounded rewards that the live web cannot.

There should be at least a small eval set based on real-world websites.

The paper already evaluates on real websites in five places:

  1. Live WebVoyager (Fig. 3): SFT on our environments raises Allrecipes from 80.0 to 85.0 and Hugging Face from 48.0 to 65.0.
  2. Online-Mind2Web, SFT (Sec. 5): capability-world training raises OM2W from 29.5 to 34.3.
  3. Online-Mind2Web, RL (Fig. 6, Table 11): 281 matched tasks on Browserbase-hosted sessions.
  4. Scaling sweeps (Fig. 4): WebVoyager and OM2W are tracked across trajectory volume and environment count.
  5. Live-web RL (Fig. 7): continuation training validated on 123 public-website tasks.

These show that gains from synthetic environments transfer to real websites. We will bring all of them together in a single table in the revision so the real-site evidence is visible in one place.

Each of the three weaknesses is answered by results already in the submission:

  • Verification is outcome-based and grounded in database state.
  • The RL recipe's gains are established on held-out tasks, with an ablation isolating its contribution.
  • Real-website evaluations appear in five places.

Combined with the strengths the reviewer already identifies, we do not believe a weak reject reflects the paper, and we ask the reviewer to raise their score.

Reviewer_7Aq8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Echoverse, a semi-automated system for building training environments for computer-use agents to augment training data. Its central idea is to jointly maintain environments, tasks, and verifiers: first, it generates applications with front and back ends, and databases from scenario specifications; then, based on agents’ execution traces, it identifies environment defects, infeasible tasks, and verification errors, and uses these findings to repair the environments or adjust the training tasks. The authors build ten complete domain environments and two environments targeting specific interaction capabilities, and use SFT and RL experiments to examine the effects of environment quality, task coverage, and stepwise rewards.

The problem addressed by the paper is practically significant, and the system implementation is fairly comprehensive. However, the evidence currently presented mainly supports the benefits of functionally complete environments, software bug fixes, and stepwise feedback for training. I still have substantial doubts about whether the proposed co-evolution mechanism represents a significant methodological contribution, and whether it can deliver consistent gains on real-world tasks at a quantifiable cost.

Critical strengths
  1. The problem selection targets real challenges in training computer-use agents. Environment defects, infeasible tasks, and faulty validation can all produce misleading training signals. A practical and valuable design choice is to incorporate these issues into a unified failure-analysis workflow and distinguish environment problems from shortcomings in the agent’s policy.

  2. The system implementation covers relatively complex and stateful workflows. Compared with generating only interfaces or static webpages, this work builds a frontend and backend, a database, task reference answers, and reset mechanisms. It includes both read and write tasks, providing infrastructure for repeated sampling and online training.

  3. Some experiments provide valuable positive evidence. On unseen widget families, joint training increased the success rate from 34.0% to 57.3% for date picker and from 62.8% to 84.8% for nested-filter. The ablation study on stepwise rewards is also persuasive, with the same initialization, training data, terminal reward, and optimization settings, mean@4 at the 20th update was 59.91% with stepwise rewards, compared with 47.19% using terminal rewards alone.

  4. The authors transparently disclose several important experimental limitations. For example, models trained before and after the ECHOSTAY fix were evaluated using the same fixed test environment and validator. The appendix also reports paired results and confidence intervals for transfering to real websites, as well as missing data in the online evaluation. These practices make the results easier to interpret.

Critical weaknesses
  1. The novelty of the core method and how it differs from prior work remain unclear. Environment synthesis, task-driven development, testing and iterative repair, state-supported verification, and process rewards have all been explored in prior research. For example, InfiniteWeb [1], which the paper already cites, includes unified specifications, task-driven testing and iterative repair, and automatically generated task evaluators. CUA-Gym [2] likewise combines the generation of tasks, environment states, and rewards, along with browser-based verification and repair workflows.

Echoverse’s most promising point of differentiation is its use of execution failures by the policy during training to continuously co-adjust the environment, tasks, and verifiers. However, Section 4 and Appendix B mainly describe how the builder, verifier, triage, and fixer work together; they do not yet adequately explain the specific methodological advance this mechanism offers over existing test-and-repair workflows. In particular, reliably distinguishing between insufficient policy capability and environment defects, and deciding what to repair and in which direction to evolve, still rely largely on model judgments and manual QA, with no independent quality measurement.

  1. The experiments on environment depth and co-evolution are confounded and do not yet provide sufficient support for mechanism-level attribution. In the comparison between shallow and deep environments, the authors changed both frontend fidelity and functional support. Appendix D.3 also notes that the training datasets for the two groups were only similar in size and were not matched at the trajectory level. Therefore, the gains on real websites may stem from improvements in visual similarity, functional coverage, task composition, and demonstration quality alike. The current results support the idea that higher-quality synthetic environments can aid training, but they do not make it possible to determine how much of the gain comes from the notion of depth as defined in this paper. The ECHOSTAY experiment has a similar issue. After the fixes, task feasibility increased from 48% to 78%, and model accuracy rose from 16.2% to 38.5%. It is reasonable that the authors compared the two models under the same test conditions. However, the original environment had severe control defects, and the distribution of feasible tasks and demonstrations available after the fixes changed as well. As a result, this experiment does not distinguish the benefits of rollout-driven co-evolution from those of ordinary software fixes and an increase in effective training data. A minor issue: A total of 37 tasks cannot yield a value of 48.0%: 17/37 ≈ 45.9%, while 18/37 ≈ 48.6%. Please clarify the actual number of evaluation tasks and how the figure was calculated.

  2. The verifier is a cornerstone of the entire framework, yet its reliability has not been quantified.

The paper emphasizes “database grounding,” but the final judgment is still made by a plain-text GPT-4.1 judge (Section 3.1, Appendix G). The paper reports no agreement rate between the judge and human annotators, nor any false-positive or false-negative rates. The prompts in Appendix G are fairly permissive: the write task explicitly accepts “additional modifications,” while the read task accepts answers containing “more information”; counts and IDs only need to be “close.” In RL, these rules could directly become reward loopholes—for example, by listing multiple candidate answers or appending write operations with side effects. The authors acknowledge this in Section 3.1, but do not conduct any reward-hacking analysis. Also, Appendix D.6 mentions that after EchoChat’s scorer was realigned, the proportion of scorable tasks rose from 34% to 99%. This suggests that the scorer may have been seriously misaligned before the fix, further underscoring the need to audit residual scoring errors in other worlds. There is also a minor issue: the scoring instructions in the read-task prompt include “0 = no required changes present…,” which looks like it was copied from the write-task prompt.

  1. The novelty of the RL component is limited, and the main results are based on in-distribution evaluation. The recipe itself combines GRPO with outcome-based rewards and LLM-judged step-by-step process rewards. Related work cited in the paper, such as PRO-CUA and VAGEN, has already explored step-by-step or process-based feedback. The paper compares only against a “no step-by-step reward” baseline, without evaluating other process-reward approaches, such as learned process reward models. The validation and training sets are drawn from the same six environments and use the same task-generation pipeline. Although the task IDs and normalized targets do not overlap, this is still an in-distribution evaluation. Most of the improvement in mean@4 occurs within the first 20 updates, after which performance plateaus. The step-by-step rewards are not centered, and their weight of 0.5 is on the same scale as the trajectory reward, so the policy is optimizing GPT-4.1’s preferences to a considerable extent. The authors should clarify whether this introduces a preference for longer responses or other biases—for example, how trajectory length changes during synthetic RL training.

  2. The gains in the synthetic environment are substantial, but strong evidence that they transfer to accuracy on real-world tasks is still lacking. Synthetic RL raises mean@4 from 39.05% to 60.34%, demonstrating that the model learns in the constructed environment. However, across 281 matched Online-Mind2Web tasks, the number of successes increases only from 154 to 159, or from 54.80% to 56.58%. Appendix F.5 reports a 95% confidence interval of −4.6 to 8.1 percentage points. The authors explicitly acknowledge this, and I appreciate their transparency; at this stage, the results are not enough to establish a reliable gain in real-world task accuracy. The clearest positive changes are a drop in the non-termination rate from 14.9% to 3.6% and a reduction in the average number of actions from 36.2 to 31.5. However, the average number of actions on successful trajectories is essentially unchanged; the overall reduction is mainly due to shorter long-tail failure trajectories. Thus, this evidence more strongly supports improved termination behavior, while any improvement in successful task execution still needs to be verified. Continued training on real websites is also preliminary: the number of successes on a fixed set of 71 questions rises from 16 to 21, but the success rate across all scorable tasks remains roughly 21.5%, and evaluation coverage varies across checkpoints. Given the distributional differences between the training tasks and the public benchmark, I would not interpret the nonsignificant results as evidence that the method is ineffective; however, the paper’s strongest evidence of gains currently remains confined to the authors’ own environment.

Questions for authors
  1. What specific gains does co-evolution provide? Compared with existing task-driven testing and repair workflows, which decisions depend on feedback from the current training strategy? Could you demonstrate the independent benefit of this feedback in a representative domain by comparing approaches with matched repair budgets and training data volumes?

  2. Can you report the agreement rate between the verifier and human annotations for each environment, including false positives and false negatives? Did the model start exploiting lenient scoring rules late in RL training—for example, by listing multiple answers or performing extra write operations?

  3. How many SFT trajectories were there in each condition in the EchoStay comparison? Does the conclusion still hold if trajectories from the repaired environment are downsampled to match the number from the original environment? Did the rollouts that triggered the repair come from the teacher model or the model being trained?

  4. In the task/reference revision comparison, did the two runs use the same version of the reference answers for validation? The mean@4 at update 0 differs between them (39.66 vs. 39.05). Is this difference due to sampling noise or a change in the reference answers? Did the revision process use trajectories from the validation set?

  5. What was the cost of building each world (in tokens or dollars, wall-clock time, and human QA hours)? What proportion of tasks were discarded in Phase 2? Were any of the 10 worlds labeled “incomplete” due to budget or plateau constraints?

Have you considered evaluating on existing controlled benchmarks such as WebArena, to enable direct comparison with other training methods?

5 comments

(1/5)

Thank you for the detailed and careful review, and for recognizing the stateful workflows, the held-out widget results, the step-reward ablation, and our reporting of limitations. We respond to each weakness and question below and list the revisions we will make at the end.

Weaknesses

The novelty of the core method and how it differs from prior work remain unclear. [...] they do not yet adequately explain the specific methodological advance this mechanism offers over existing test-and-repair workflows.

Echoverse differs from build-time test-and-repair workflows in where the repair signal comes from and when repair stops. In a fixed-task, test-driven loop, repair is driven by the builder's own tests and ends when they pass. In Echoverse, repair continues after a world ships. Failures in training rollouts are triaged with white-box access to code, database, and browser, and the environment, tasks, and verifier are repaired together. Concretely, Echoverse adds three things:

  • Rollout-driven repair after shipping: defects that build-time checks miss surface through training failures. The EchoStay guest-count defect was found this way.
  • Task-relative depth: a world is accepted only when the workflows required by its task corpus function end to end, not when a single execution succeeds (Sec. 3.2).
  • Database-grounded references: references are SQL queries over the seeded database, and write tasks are graded on the sqldiff between initial and final state.

We compare directly against the test-driven workflow. Our shallow baseline in Sec. 5 follows InfiniteWeb's fixed-task, test-driven loop (Zhang et al., 2026). With comparably sized corpora, it yields no gain or a loss on live WebVoyager (80.0 → 75.0 and 48.6 → 48.6), while Echoverse worlds yield 85.0 and 64.9. We will sharpen this positioning in Sec. 4.

In particular, reliably distinguishing between insufficient policy capability and environment defects, and deciding what to repair and in which direction to evolve, still rely largely on model judgments and manual QA, with no independent quality measurement.

We agree that triage relies on model judgments and sampled manual QA. Three safeguards constrain it:

  1. White-box triage: triage agents check whether the goal's entities exist, whether the required workflow is reachable, and whether the verifier checks the intended outcome (App. B.5). They inspect code and database state directly rather than inferring from screenshots.
  2. Goal preservation: a repair must preserve the task goal. Weakening a task to obtain a passing result is treated as a regression.
  3. Withholding: ambiguous cases are withheld until further inspection rather than labeled as policy or world failures.

For an independent measurement of the verifier, we will add a human agreement study on 100 trajectories for the camera-ready (see Q2).

In the comparison between shallow and deep environments, the authors changed both frontend fidelity and functional support. [...] they do not make it possible to determine how much of the gain comes from the notion of depth as defined in this paper.

We agree. The study compares two pipelines rather than isolating depth from frontend fidelity, and we will state the claim that way. The two corpora cover the same domains, are comparably sized, and share the same alignment data. The comparison therefore establishes that worlds from our pipeline transfer better than worlds from a fixed-task, test-driven pipeline. It does not attribute the gain to depth alone. We believe the pipeline-level comparison is still the one that matters for practitioners deciding how to build training environments.

(2/5)

Weaknesses (continued)

As a result, this experiment does not distinguish the benefits of rollout-driven co-evolution from those of ordinary software fixes and an increase in effective training data.

On training data, both conditions use roughly 1,000 trajectories, within 10% of each other, and both checkpoints are evaluated on the same repaired world with the same verifier.

On the shift in feasible tasks: this is the intended effect, not a confound. The repairs made blocked workflows trainable, so the gain reflects training on workflows that were previously broken.

On software fixes: we do not claim the fixes themselves differ from ordinary software fixes. Co-evolution is how the defects were found. The guest-count defect was missed by build-time checks and surfaced through failed bookings in both GPT-5.4 and trained-policy rollouts.

We also need to correct how the paper reports EchoStay feasibility. The 48% → 78% change comes from an earlier repair round of six fixes on a 100-task test set (46/95 → 78/100; five tasks timed out or crashed in the initial run). The guest-count fix came in a later round and recovered 15 of 24 affected bookings. We will correct Sec. 5 and App. D.6.

A minor issue: A total of 37 tasks cannot yield a value of 48.0%

The reviewer is right. The 37-task set is Hugging Face (App. D.3), and the reported 48.0 is a transcription error: the correct value is 48.6 (18/37). The deep-environment score of 65.0 should likewise read 64.9 (24/37). We will correct both in the abstract, body, and Figure 3. Neither correction changes the conclusions.

The paper reports no agreement rate between the judge and human annotators, nor any false-positive or false-negative rates.

The judge's task is narrow. It compares the agent's answer, or the sqldiff between initial and final database state, against a SQL-derived reference. It never sees screenshots or action history, so it does not have to infer what happened.

We have conducted manual agreement reviews on sampled verifier judgments and are confident in their accuracy, but did not record formal agreement rates. For the camera-ready, we will add a human agreement study on 100 trajectories, reporting false-positive and false-negative rates.

The prompts in Appendix G are fairly permissive [...] In RL, these rules could directly become reward loopholes—for example, by listing multiple candidate answers or appending write operations with side effects.

Each permissive rule addresses a specific source of false negatives:

  • Extra writes: applications log incidental updates even when the agent only browses.
  • More information: agents often add context alongside the core answer, such as a date with a time.
  • Close counts and IDs: values displayed in the UI are rounded and paginated differently from the database.

Strict matching would penalize correct behavior in each case.

We did not run a dedicated reward-hacking analysis, but several properties of the setup limit the room for exploitation:

  • Synthetic RL tasks are mostly write-oriented. For these, the outcome judge sees only the reference and the database diff, so answer text cannot affect the reward.
  • The policy runs without a thinking block and its per-step reasoning is short, so response length is not an optimization target.
  • Extra writes earn no reward on their own. The step judge scores each action from screenshots and database changes, so unhelpful writes receive no step credit, and repeated loops are penalized.
  • When co-evolution finds a class of false positives, we add constraints to the reference that disallow it.
  • Validation rollout length stays nearly flat over training (21.0 → 22.2 turns). This is not what we would expect if the policy were systematically adding extra writes.

Any remaining exploitation of this kind would appear as false positives in the camera-ready agreement study.

(3/5)

Weaknesses (continued)

Appendix D.6 mentions that after EchoChat's scorer was realigned, the proportion of scorable tasks rose from 34% to 99%. This suggests that the scorer may have been seriously misaligned before the fix, further underscoring the need to audit residual scoring errors in other worlds.

Yes, the EchoChat verifier was misaligned before the fix. This is the co-evolution loop working as intended: rollout analysis exposed the misalignment, and the verifier was realigned with the database. EchoChat is not part of the RL training set. Every world must pass the 95% acceptance thresholds on claims and tasks before use. The camera-ready agreement study is our audit of residual scoring errors.

There is also a minor issue: the scoring instructions in the read-task prompt include "0 = no required changes present…," which looks like it was copied from the write-task prompt.

Thank you for catching this. The read judge's score guide mislabels the 0 case with text copied from the write judge; the score is binary and the pass condition is stated correctly, so 0 still denotes any answer that is not semantically correct.

The recipe itself combines GRPO with outcome-based rewards and LLM-judged step-by-step process rewards. Related work cited in the paper, such as PRO-CUA and VAGEN, has already explored step-by-step or process-based feedback.

We do not claim process rewards as new. Our contribution is that database-grounded, resettable worlds make outcome-plus-step supervision cheap and grounded:

  • Cost: synthetic RL needs about 2 judge requests per trajectory, versus 47 on the live web, and provides exact resets.
  • Grounding: our step judge sees database changes alongside screenshots, so action-level credit is tied to actual state changes. PRO-CUA optimizes candidate actions with a process reward model, and VAGEN supplies turn-level world-modeling rewards. Our step judge instead scores executed actions against database changes and task progress.

The paper compares only against a "no step-by-step reward" baseline, without evaluating other process-reward approaches, such as learned process reward models.

We agree. Our ablation answers a narrower question: whether grounded action-level feedback helps beyond outcome supervision in these environments. With initialization, training data, trajectory rewards, and optimization settings held fixed, it adds 12.73 pp of mean@4 at update 20. Comparison against learned process reward models is a good direction, and we will note it as a limitation.

Although the task IDs and normalized targets do not overlap, this is still an in-distribution evaluation.

We agree the 290 validation tasks share environments with training. Out-of-distribution evidence comes from three sources:

  • held-out widget families never seen in training (34.0 → 57.3 and 62.8 → 84.8)
  • live WebVoyager
  • Online-Mind2Web

Most of the improvement in mean@4 occurs within the first 20 updates, after which performance plateaus.

This is expected with a fixed task pool. Training tasks are selected because the initial policy solves them one to three times out of four. As the policy improves, more groups become all-success, and their centered outcome advantage goes to zero, so the outcome signal shrinks. Pass@4 reaches 80.0% by update 20. Performance holds through update 80 without degradation, and we use that export for the transfer experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.