acceptodds
Under review as a conference paper at ICLR 2027

EnvCraft: Synthesizing Executable Environments in Agentic RL for Claw-like Agent

10 views

Abstract

The paradigm of LLMs has rapidly shifted from passive language interfaces to autonomous Claw-like agents that execute long-horizon tasks across stateful workspaces. While Agentic Reinforcement Learning (Agentic RL) provides a promising path to optimize these agents, its scaling is heavily bottlenecked by the severe scarcity of interactive training environments. Existing synthetic environments are strictly limited to tool-calling endpoints, rendering them insufficient for accommodating the end-to-end real-world demands of claw-like agents. To bridge this gap, we introduce ClawForge, an automated framework for synthesizing executable environments and scalable training data. Specifically, ClawForge employs an environment synthesis engine to build sandbox-isolated workspaces, alongside a topology-aware data generation engine to produce coherent task trajectories. Overall, we synthesize 139 interactive environments comprising approximately 20K complex tasks for Agentic RL training. Experiments on Qwen3/3.5 models (8B–32B) show that ClawForge yields gains of up to +11.9% on Claw-style benchmarks and +8.0% on general tool-use benchmarks, with concurrent reductions in inference token cost. The result confirms that synthesized executable environments

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Environments, RL, Harness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PVVjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EnvCraft, an automated framework that synthesizes executable sandbox environments and RL training tasks for Claw-like agents. LLMs expand 83 natural-language scenarios into environment code guided by nine interaction archetypes, and Claw environments additionally receive OpenClaw's built-in bash tools and generated skill files. Tasks are built by sampling tool chains from a bi-level tool/environment dependency graph, reverse-engineering multi-turn user intents, and generating initial states with deterministic verifiers, yielding 139 environments and 19,777 tasks. Trained with GRPO on these tasks, Qwen3-8B improves from 12.94 to 24.85 on PinchBench and from 44.06 to 55.42 on Claw-Eval. Gains also appear on BFCL-v3 and, to a smaller extent, τ²-bench.

Critical strengths
  1. This paper proposes a practical framework that could truly help to address the problem of limited environments and training tasks for claw-like agents.

  2. Training on the synthesized tasks gives clear gains on Claw-style benchmarks for three backbones while lowering token usage.

Critical weaknesses
  1. Insufficient baselines. Every controlled comparison is between a base model and the same model after RL on EnvCraft data, which does not show that EnvCraft's environments are better than existing ones. Prior synthesis methods (EnvScaler [1], ScaleEnv [2], AWM [3], EnvFactory [4]) appear only as numbers copied from their papers under different recipes, and none is evaluated on PinchBench or Claw-Eval. At 8B, ScaleEnv, AWM and EnvFactory report 38.53, 34.40 and 33.67 on τ²-bench vs. 29.80 for EnvCraft, and EnvFactory reports 49.00 on BFCL-v3 vs. 46.25, these methods all outperform EnvCraft's results.

  2. The novelty is limited. Prior work listed in Table 1 already covers the main steps of the pipeline. The Claw-specific additions are mainly the injected OpenClaw bash/file tools and the co-generated skill files. As shown in Fig. 11, the Claw gain appears only after these tools are added, while Claw environments without them score close to general tool data. The main new ingredient therefore seems to be training with the evaluation harness's own tool interface rather than the synthesis pipeline itself.

[1] Song X, Chang H, Dong G, et al. EnvScaler: Scaling tool-interactive environments for LLM agent via programmatic synthesis[C]//Findings of the Association for Computational Linguistics: ACL 2026. 2026.

[2] Tu D, Hao H, Yang H, et al. ScaleEnv: Scaling environment synthesis from scratch for generalist interactive tool-use agent training[C]//International Conference on Machine Learning. 2026.

[3] Wang Z, Xu C, Liu B, et al. Agent world model: Infinity synthetic environments for agentic reinforcement learning[J]. arXiv preprint arXiv:2602.10090, 2026.

[4] Xu M, Wang Z, Deng M, et al. EnvFactory: Scaling tool-use agents via executable environments synthesis and robust RL[J]. arXiv preprint arXiv:2605.18703, 2026.

Questions for authors
  1. Can you train the same backbone under the same RL budget on EnvScaler [1] or AWM [3] data with the same OpenClaw built-in tools added, and report PinchBench and Claw-Eval results?

  2. Can you clarify which parts of the pipeline are new relative to ScaleEnv [2] and EnvFactory [4], and ablate the components left to future work (interaction archetypes, intent decomposition, personas)?

  3. From my experience, the reported results on these benchmark are sometimes noisy since the number of problems in the benchmark are small and the tasks are sometimes edge cases. Please report the number of tasks per benchmark and avg@k (e.g., k = 4, 8, 16).

[1] Song X, Chang H, Dong G, et al. EnvScaler: Scaling tool-interactive environments for LLM agent via programmatic synthesis[C]//Findings of the Association for Computational Linguistics: ACL 2026. 2026.

[2] Tu D, Hao H, Yang H, et al. ScaleEnv: Scaling environment synthesis from scratch for generalist interactive tool-use agent training[C]//International Conference on Machine Learning. 2026.

[3] Wang Z, Xu C, Liu B, et al. Agent world model: Infinity synthetic environments for agentic reinforcement learning[J]. arXiv preprint arXiv:2602.10090, 2026.

[4] Xu M, Wang Z, Deng M, et al. EnvFactory: Scaling tool-use agents via executable environments synthesis and robust RL[J]. arXiv preprint arXiv:2605.18703, 2026.

Reviewer_Y1noOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes EnvCraft, a framework for automatically constructing executable environments and training tasks for Claw-like agents. Its main components are an environment synthesis engine that generates stateful workspaces with tool interfaces, file-system access, and shell execution, and a task synthesis engine that uses a two-level tool–environment dependency graph to construct multi-step workflows. These workflows guide the generation of user instructions, initial states, and programmatic verifiers that provide rewards for reinforcement learning. The pipeline also includes environment validation, task auditing, and difficulty filtering. The authors generate 139 environments and 19,777 tasks and use GRPO to train three Qwen3/Qwen3.5 model configurations ranging from 8B to 32B parameters. Evaluation covers workspace-oriented tasks on PinchBench and Claw-Eval, together with general tool use on BFCL-v3 and τ²-bench. The reported results show improvements over the corresponding base models, with Claw-specific training also reducing inference token usage. Additional analyses examine the contributions of tool-chain sampling and workspace extensions, as well as verifier reliability and potential training–test overlap.

Critical strengths
  1. The paper addresses a practical bottleneck in agent training: constructing executable environments that support file operations, shell execution, and multi-step workflows.

  2. The framework provides a complete pipeline covering environment construction, task generation, verification, and quality filtering, producing a substantial collection of training resources.

  3. Claw-specific training improves performance across three model configurations while reducing inference token usage, providing evidence of the practical value of the generated data.

Critical weaknesses
  1. Limited methodological novelty. The framework closely resembles EnvFactory in its executable-environment synthesis, dependency-based task generation, and multi-turn interaction design. Its main distinction appears to be the adaptation to Claw-style workspaces, while the technical advances beyond this adaptation and integration remain insufficiently established.

  2. Insufficient controlled comparisons. Comparisons with existing environment-synthesis methods use different training recipes and budgets. The results therefore demonstrate the usefulness of EnvCraft-generated data, but do not establish whether its synthesis approach is more effective than existing alternatives under comparable conditions.

  3. Unclear attribution of gains. Key ablations do not adequately separate design improvements from differences in training data. In particular, the topology-sampling comparison does not match the number of valid tasks retained after filtering, leaving unclear how much of the improvement comes from better task construction versus a larger effective training set.

  4. Unresolved reward reliability. The verifier audit identifies both false acceptance and false rejection, yet their effects on RL training are not examined. Since these verifiers provide the optimization signal, further analysis is needed to establish whether training amplifies verification errors or encourages exploitation of incomplete success criteria.

Questions for authors
  1. What are the main technical advances over EnvFactory, and which challenges specific to Claw-style tasks require these changes?

  2. Can you provide a controlled comparison with a representative environment-synthesis method using the same backbone, training recipe, and budget?

  3. Does topology-aware sampling still outperform the alternatives when the number of valid tasks retained after filtering and the RL training budget are matched?

  4. How do the observed verifier errors affect RL training? Have you checked whether the policy exploits false-accept cases or whether correcting these errors changes the results?

Reviewer_9DaXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents EnvCraft, an automated pipeline that builds executable training environments and verifiable tasks for reinforcement learning of "Claw-like" agents, which work on files, shells and databases through the OpenClaw harness, as well as general tool-use agents. An LLM turns scenario descriptions into sandboxed environments, composes tasks from chains of dependent tools, and writes matching user requests and verification scripts, with automatic checks discarding broken outputs. Training Qwen models with RL on this data improves their scores on Claw-style benchmarks while using fewer tokens, and also helps on general tool-use benchmarks.

Critical strengths
  1. It is a practical, cheap pipeline for a timely target. Agents that work in persistent workspaces through tools like OpenClaw are a new need, and there are very few training environments for them. EnvCraft makes over a hundred environments and about 20K checkable tasks for a few hundred dollars of API cost, with no hand-built sandboxes. Some design choices are smart: tasks always end in a state change so rewards can be checked, user requests hide in-between values, and the request, workspace and checker are made together so they stay consistent. If it's released, others could use it.

  2. The in-domain gains are real, and the paper does some honest checks. Claw-specific training about doubles the 8B model's PinchBench score while using fewer tokens. General tool-use training alone does little, which suggests the workspace-style tasks matter. The paper also has a human check of the verifiers, a sampling ablation, and a decontamination check against BFCL that openly shows part of the BFCL gain going away.

Critical weaknesses

It isn't shown that the synthesized environments drive the Claw gains, or that EnvCraft beats other synthesis methods.

  • The paper's own ablation shows that training on the synthesized environments without OpenClaw's built-in bash, read and write tools barely helps. Most of the gain shows up once those tools are added, and they are exactly the tools of the evaluation harness. So the headline gain may mostly be "RL inside the OpenClaw tool interface". A control with simple OpenClaw tasks and no synthesis pipeline would separate the two.
  • Baselines are numbers copied from other papers with different recipes, and there is no baseline at all on the Claw benchmarks. Even the copied numbers don't always favor EnvCraft. Several prior methods report higher τ²-bench scores, and EnvFactory reports a higher BFCL-v3 overall score than EnvCraft-Tool at 8B. That contradicts the text, which says τ²-bench is the only benchmark where prior methods look better.
  • The paper argues for large-scale synthesis, but there is no scaling analysis, and training uses only part of the corpus. Most components (templates, rubric, difficulty filter, personas) aren't ablated, and everything is a single run.

Overlap with the test benchmarks isn't ruled out, and some reported settings don't match.

  • The Claw scenarios come from OpenClaw community resources, the same place the Claw benchmarks come from. But the decontamination check only covers BFCL, not PinchBench or Claw-Eval, where the headline gains are.
  • The general scenarios are partly taken from BFCL itself and closely mirror its multi-turn API suites. Removing the flagged environments takes away about a third of the BFCL gain. So the claim that the gains aren't limited to Claw-style tests should be softer.
  • Some settings disagree: the turn limit (64 vs. 32), the number of evaluation runs (three vs. eight), and the full-setup PinchBench score (24.33 vs. 24.85). The standard deviations also look too small for multi-turn agents sampled at temperature 1.0, and training-seed variance isn't reported.
Questions for authors
  1. Did you check overlap between your Claw scenarios and tasks and the PinchBench / Claw-Eval task sets? What happens to the gains after removing anything flagged?

  2. How much gain remains with a harness-only control, meaning RL on simple OpenClaw tasks (or on another method's environments) with the same tools and budget?

  3. Can you train the same model with the same budget on at least one public prior synthesis corpus, so the comparison is fair?

  4. Which numbers are right for the turn limit, the number of evaluation runs and the full-setup score? How many tasks are in PinchBench and Claw-Eval, and what do the ± values vary over?

  5. How many tasks and RL steps were used for direct Claw training? How does performance change with more environments or tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.