acceptodds
Under review as a conference paper at ICLR 2027

CompoWorld: Compositional Environment Scaling for General Agents

Abstract

Automatically generated environments provide a scalable source of interaction data for training general agents. However, existing approaches mainly generate tasks within a single environment, while real-world workflows require agents to connect information and actions across multiple services. We introduce Compositional Environment Scaling (CompoWorld), which expands the task space by composing a finite library of reusable services. Coding agents turn tool specifications into verified services with typed states and shared interfaces, while a world model handles tools that cannot be reliably implemented. A random-walk procedure connects services through dependency graphs, enabling the generation and verification of tasks that require information to flow across services. Verified trajectories support supervised fine-tuning (SFT), while our Completion-Focused Rubric Reward guides reinforcement learning (RL) toward full task completion by emphasizing criteria with lower pass rates within each rollout group. We construct 448 services exposing 10,130 tools and use 3K SFT trajectories and 1K RL tasks to train Qwen3.6-35B-A3B. Experimental results show that CompoWorld improves on its backbone by 9.17 points on average across eight benchmarks. On AutomationBench, it surpasses frontier models such as Claude Opus 4.6 and leads all compared agent-specialized 35B-A3B models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
environment scaling, llm agents, large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sVm9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CompoWorld, a framework that composes reusable software services into verifiable cross-service tasks for LLM agent training. It combines automated environment synthesis, dependency-based task generation, SFT, and a completion-focused RL reward. Experiments on eight benchmarks show an average improvement of 9.17 points over the Qwen3.6-35B-A3B backbone.

Critical strengths
  1. Practical framework: The reusable service library and automated task verification provide a promising approach to scalable agent training.
  2. Encouraging results: The single-versus-composed environment comparison suggests that cross-service tasks can improve agent performance under a fixed data budget.
Critical weaknesses
  1. Weak compositional dependency modeling. Random walks do not guarantee meaningful cross-service reasoning. The generated tasks may primarily increase tool-call length rather than require genuine compositional generalization.
  2. Limitations of the proposed reward. Completion-Focused Rubric Reward relies on pass-rate estimates from only eight rollouts and provides no discrimination when all rollouts fail a criterion. Its advantage over uniform rubric rewards remains insufficiently established.
Questions for authors
  1. How do you ensure that generated dependencies require genuine cross-service reasoning?
  2. How robust is the proposed reward to rollout-group size and universally failed criteria?
2 comments

Thank you very much for your evaluation of our work and your constructive comments.

W1&Q1: We would like to clarify that Random Walk does not simply randomly concatenate services; rather, it is used to construct cross-service workflows with explicit data dependencies and state dependencies. Specifically, the inputs required by a downstream service must come from the execution results (Observation) of preceding services and cannot be obtained independently. In addition, we require that at least one service be visited repeatedly, while avoiding consecutive visits to the same service, thereby introducing dependencies caused by state changes—for example, the Agent needs to revisit a service after operations on other services in order to obtain the updated state.

Figure 5(a) further verifies the value of compositional-environment training: under the conditions that the number of training trajectories (1K), the number of environments, and the SFT pipeline are kept consistent, compositional-environment training achieves better results on all four benchmarks. Appendix I also provides complete cross-service execution trajectories as concrete examples.

W2&Q2: We would like to clarify that, for Criteria that no Rollouts pass, our method indeed cannot further distinguish between different trajectories. This is essentially the same as in GRPO: when all Rollouts receive the same reward, the within-group relative advantage is zero, and no effective optimization signal can be provided.

Regarding Rollout Group Size, our method adopts the same within-group sampling and relative estimation mechanism as GRPO, so its sensitivity to Group Size has the same nature. Increasing the number of Rollouts can improve the stability of completion-rate estimation, but it also increases sampling cost. Group Size = 8 is a common setting for GRPO.

The core advantage of our method lies in the fine-grained reward signal. For Criteria where some Rollouts succeed and some fail, we increase their relative weight so that the model focuses on optimizing capabilities that have not yet been stably mastered; for Criteria where all Rollouts succeed or all fail, no additional discriminative signal is produced. Therefore, our method essentially refines the within-group relative optimization in GRPO to the Criterion level, and thus will certainly provide stronger supervision signals than GRPO.

Thank you again for your constructive feedback. We have provided detailed responses to your concerns regarding compositional dependencies and the completion-focused reward.

We would greatly appreciate it if you could take another look at our rebuttal and let us know whether it addresses your concerns. If any questions remain, we would be happy to provide further clarification.

Reviewer_4kLLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CompoWorld, a framework for scaling agent training environments through the composition of reusable services. It first constructs executable services from tool specifications, combining verified implementations with world-model simulation for tools that cannot be reliably implemented. It then generates cross-environment tasks using dependency graphs and verifies their feasibility through executable task probes. The resulting trajectories are used for supervised fine-tuning, followed by reinforcement learning with a Completion-Focused Rubric Reward that assigns higher weights to less frequently satisfied criteria. The authors train Qwen3.6-35B-A3B using 3K SFT trajectories and 1K RL tasks and evaluate the resulting agent across eight benchmarks. The results show an average improvement of 9.17 points over the base model, with particularly large gains on AutomationBench and SkillsBench.

Critical strengths
  1. The paper proposes a scalable approach to generating cross-environment tasks by composing reusable services through dependency graphs, allowing the construction of complex workflows without generating new services for each task.

  2. The proposed Completion-Focused Rubric Reward introduces an adaptive weighting mechanism that prioritizes criteria with lower completion rates, providing a more targeted reward signal for reinforcement learning.

  3. The paper evaluates the trained agent across eight benchmarks covering diverse agent capabilities and provides additional analyses of environment scaling, single-environment versus composed-environment training, and the contribution of reinforcement learning.

Critical weaknesses

The experimental results are promising. However, several concerns and questions remain regarding the novelty of the method and the reported improvements. I would appreciate the authors' responses to the following points.

  1. The proposed method lacks clear novelty. The first stage of environment synthesis (Figure 2(a)) has already been explored in previous works, such as VULCAN, EnvScaler, and AgentScaler. Furthermore, the authors combine synthesized executable environments with world-model simulation, which raises questions about the reliability of the generated environments. In particular, how reliable are these synthesized environments, and how much of the performance improvement comes from verified executable environments versus world-model simulation? An ablation study separating these components would help clarify their individual contributions.

  2. CompoWorld constructs 448 services exposing 10,130 tools, which raises concerns about potential data contamination. Given the scale of the generated environments and tasks, it is important to investigate whether the training tasks overlap with the evaluation benchmarks. I suggest reporting cosine similarity between training and benchmark tasks to provide stronger evidence that the improvements are not primarily driven by data contamination.

  3. Figure 4(a) shows that training on single-environment tasks decreases performance on τ³-Banking and DeepPlanning compared with the base model. This raises concerns about the quality and effectiveness of the generated training data. The authors should provide a clear justification for why training on these tasks leads to worse performance than the base model.

  4. Please include detailed results for all benchmarks, such as Vita-Benchmarks. Also share some training trajectories to provide a sense of the training data.

  5. The work heavily relies on Qwen3.6-35B-A3B. What about other open-source models such as Qwen3.5-9B or other backbones?

Finally, the reported performance improvements are surprisingly large, but the paper provides limited explanation for why the proposed method achieves such substantial gains compared with previous approaches. In particular, CompoWorld surpasses several frontier models on AutomationBench, while previous environment-synthesis methods have not reported comparable improvements. It remains unclear which components of the proposed method are responsible for these gains and what distinguishes this approach from previous methods at the experimental level. A more detailed analysis of the results and comparisons with existing environment-synthesis methods would help explain these improvements.


References:

VULCAN: https://openreview.net/forum?id=Ht4HB4pIQq&noteId=SS4dHupwle - https://github.com/microsoft/vulcan

EnvScaler: https://arxiv.org/abs/2601.05808

AgentScaler: https://arxiv.org/abs/2509.13311

Questions for authors

Please read the weaknesses.

6 comments

We sincerely thank you for your evaluation of our work and your constructive comments.

W1: We thank the reviewer for pointing out related work such as VULCAN, EnvScaler, and AgentScaler. We acknowledge the important contributions of these studies to automated environment construction. We wish to clarify that the main contribution of CompoWorld is not the proposal of an entirely new single-environment synthesis algorithm, but rather the extension of the basic unit of environment construction from independent services to compositional environments with cross-service dependencies.

In addition, CompoWorld adopts different considerations in environment implementation. Our environments are organized around complete services, and a service typically contains multiple tools that must work cooperatively. If certain core tools are not implemented, the entire service may not function properly. Therefore, we introduce World Model Simulation to supplement long-tail tools that are difficult to implement reliably through local code, such as operations that depend on external services, model inference, or special execution environments. World Model Simulation is not a replacement for executable environments but serves to supplement tool coverage.

According to the statistics in Appendix B, only about 20% of services contain tools that depend on World Model Simulation; tools that actually rely on the World Model account for only 7.1%, and about 10% of training samples invoke a simulated tool at least once. Therefore, the vast majority of tools and training samples are still based on tested deterministic code execution. We also analyzed the relevant training trajectories and found that although the simulation results of the World Model exhibit a certain degree of inconsistency, they are generally of good usability. Because it covers only a small number of tools and removing the simulation mechanism may prevent some services from being executed completely, a strict independent ablation is difficult to conduct. We believe that the main role of the World Model is to ensure the integrity and usability of the environment, while the main training benefits of CompoWorld still come from large-scale, executable compositional environments.

W2: We compared the similarity between the training data and public benchmark tasks, added a t-SNE visualization in Figure 7 of the revised manuscript, and provided further analysis in Appendix G.

The results show that the training tasks have broad coverage, and no samples completely identical to the evaluation tasks were found. We note that some CompoWorld tasks share certain similarities with AutomationBench in terms of business scenarios and task objectives, but there are clear differences in specific service compositions, tool invocation paths, and execution workflows. Therefore, we believe there is no direct data contamination.

W3: We believe that this phenomenon is mainly related to the complexity gap between single-environment tasks and compositional-environment tasks. We additionally calculated the average trajectory length of the two task types: single-environment tasks have 7,279 tokens, whereas multi-environment tasks have 41,860 tokens, approximately 5.8 times the former. Single-environment tasks usually involve shorter tool invocation chains, whereas compositional-environment tasks require handling cross-service information transfer, multi-step execution, and complex state dependencies. We conjecture that the relatively simple training distribution of single-environment tasks may make the model more inclined to learn short-range tool invocation patterns, thereby causing negative transfer to certain benchmarks that require long-range planning and complex reasoning. At the same time, we believe that task difficulty cannot be completely decoupled from environment compositionality. Compositionality itself is an important source of task complexity, and existing single-environment synthesis methods can hardly construct tasks of comparable complexity. This also reflects the advantage of CompoWorld in constructing complex agent tasks through cross-service composition.

W4: Regarding the benchmark results, VitaBench and VitaBench 2.0 are reported separately and have not been merged into a single metric. In addition, Table 2 provides fine-grained results for the six business domains of AutomationBench (Finance, HR, Marketing, Operations, Sales, and Support).

We further provide detailed results for each business domain of VitaBench:

Business DomainBackbone Avg. ScoreCompoWorld Avg. ScoreImprovementBackbone Pass@4CompoWorld Pass@4
Cross-domain28.0038.50+10.505470
Delivery48.5056.00+7.508186
In-store53.0060.00+7.008886
OTA26.2543.25+17.005179
Overall38.9449.44+10.5068.5080.25

Regarding the training trajectories, we have provided a complete end-to-end execution trajectory in Appendix I (A Complete Agent Trajectory). It demonstrates a cross-service task involving Google Calendar, an email and collaboration service (Simosphere), Reddit, and Thinair Geo. In addition, Appendix H provides the typed state definition and concrete tool implementation examples for the Google Calendar environment.

W5: We clarify that our choice of Qwen3.6-35B-A3B as the main experimental backbone is mainly based on the characteristics of the tasks in this paper. Tasks generated by CompoWorld typically involve cross-service information transfer, multi-step tool invocation and the joint satisfaction of multiple task constraints, thus requiring the model to have strong foundational agent capabilities. We aim to focus on the performance improvements brought by the compositional environment and training method while minimizing the model's own capability bottlenecks as much as possible. For smaller models such as Qwen3.5-9B, directly transferring the current training setup would also require considering the capability gap between the teacher model and the student model (for example, Qwen3.5-9B scores approximately 0 on AutomationBench). Our training trajectories are generated by a DeepSeek v4 teacher model, and a large capability gap may affect the learning effectiveness of smaller models on complex trajectories, as our data are not well suited to such models. In summary, we respectfully do not consider additional comparative experiments with smaller models to be a reasonable requirement.

Finally:

We believe that the significant performance improvement of CompoWorld mainly comes from the following aspects:

  • Task diversity: CompoWorld not only constructs independent services but also composes multiple services to generate complex tasks with cross-service dependencies, significantly expanding the coverage of training tasks.
  • Task complexity: Compositional environments bring longer execution chains and more complex state dependencies, providing the model with richer training signals for long-range planning and tool collaboration.
  • Trajectory quality: We use a teacher model with strong agent capabilities to generate high-quality execution trajectories, enabling tasks in complex environments to be converted into effective training data.

Our model exhibits a substantial improvement in capability. This should be regarded as a strength rather than a point of weakness. We will release the model for verification.

Thank you for the additional explanations. However, I am still not convinced that the proposed training data is effective beyond Qwen3.6-35B-A3B. I believe the paper needs further experiments across different models to demonstrate the generalizability of the training data quality. Therefore, I have decided to maintain my original score.

Reviewer_zF2UOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CompoWorld generates agent training tasks by composing reusable, stateful services through cross-service dependencies. The framework combines tested tool implementations with selective world-model simulation, verifies generated tasks through executable probes, and introduces a rubric reward that emphasizes criteria with lower group pass rates. Training Qwen3.6-35B-A3B with 3K SFT trajectories and 1K RL tasks improves performance by 9.17 points on average across eight benchmarks.

Critical strengths

The framework provides a practical way to reuse executable services across many training tasks. Constructing 448 services with 10,130 tools is a substantial effort, and the resulting task generation pipeline could be useful beyond the particular model studied here.

The empirical gains are substantial and extend across eight benchmarks. The Qwen3.6-35B-A3B model used is also large enough to demonstrate the effectiveness of the proposed method.

Critical weaknesses

The experiments don't directly demonstrate systematic compositional generalization. The 1K comparison matches trajectory count, but the paper does not show whether the datasets are comparable in token count, interaction length, or constraint complexity. The results support the usefulness of the training recipe, but leave open how much of the gain comes specifically from learning cross-service dependencies.

The role of cross-service dependencies is not sufficiently isolated. Section 3.2 constructs dependencies and verifies that reference solutions execute successfully, but does not test whether these dependencies are necessary or can be bypassed. Figure 4(a) provides useful evidence by matching the number of training trajectories, although it does not report whether training tokens and tool-call counts are comparable. A baseline that combines independent subtasks without cross-service dependencies would help separate the benefit of dependency structure from longer, more complex trajectories. The evaluation also lacks a controlled split that tests unseen service combinations or dependency structures while using services already seen during training. This would test whether the model can combine its existing capabilities to complete new tasks.

Questions for authors

For Figure 4(a), how do the two training sets compare in total tokens, tool calls, and number of task constraints? Were they generated using the same teacher and filtering procedure?

Are there results on held-out service combinations or dependency structures built from familiar services?

Starting from the same SFT checkpoint and using the same rollout budget, how do uniform and completion-focused rewards compare on held-out full-task success? Please also clarify the evaluation set and score used in Figure 6.

The paper only states "3K successful trajectories for SFT, followed by 1K tasks for RL", without clarifying whether these 1K RL tasks are a subset of the SFT tasks or whether the two sets are completely disjoint.

1 comment

Thank you very much for your positive evaluation of our work and for your constructive comments.

W1 & Q1: First, we would like to clarify that the comparison between Single-Environment and Composed-Environment Training in the paper already controls for the following factors: both groups use 1K training trajectories; both groups use the same number of training environments; and both groups use the same backbone and SFT training pipeline. This design aims to rule out, as much as possible, the effects of training sample count, environment count, and training configuration, so as to study the differences brought by the way environments are composed.

Regarding the trajectory length and token counts you mentioned, we have added additional statistics: on average, a single-environment trajectory contains 7,279 tokens, while a multi-environment task contains 41,860 tokens. This indicates that our composed tasks are more complex and require longer trajectories. We need to clarify that, for Single-Environment versus Composed-Environment, task difficulty and compositionality cannot be fully decoupled; these tasks are harder precisely because compositionality increases task complexity. Existing single-environment synthesis methods cannot construct tasks of comparable difficulty. Therefore, we believe that the benefits brought by cross-service dependencies are mainly reflected in increased task difficulty, longer-horizon requirements, and greater task diversity.

W2 & Q2: We believe that the baseline without cross-service dependencies proposed by the reviewer is very valuable. Because constructing tasks, trajectories, and training models is very costly, it is very difficult to additionally construct and implement a baseline “constructed by concatenating independent subtasks without cross-service dependencies” during the rebuttal period. In addition, one problem with concatenating independent subtasks is that the constructed tasks are not sufficiently natural or realistic. The advantage of our cross-environment tasks is that they are closer to real-world scenarios. We will supplement the discussion and experiments in our paper.

Q3: Because the full test scale is too large, the per-checkpoint testing in Figure 6 uses an independent 500-question validation set; this validation set is drawn from the same source as our constructed tasks and is in-domain. On this validation set, the completion-focused reward has an advantage. We did not use the checkpoint reported in the main table for per-checkpoint testing because the full evaluation of each checkpoint is very expensive. Therefore, the final results report the final RL checkpoint in Table 1, and the pure SFT results are shown in Figure 3.

Q4: We clarify that the 1K data used for RL and the 3K SFT data have no overlap.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for overseeing the review process.

We would like to highlight several important updates in our rebuttal:

  1. To address Reviewer 4kLL's concern, we conducted an additional experiment on Qwen3.5-9B. Fine-tuning on 3K samples improved its AutomationBench score from 29.29 to 63.34 (+34.06 points), with consistent gains across all six domains. Its full-task pass rate also increased from 5.17% to 23.67%. This provides further evidence that our training data is effective beyond the original 35B backbone.

  2. We clarified how CompoWorld constructs explicit cross-service information and state dependencies, and provided additional trajectory analysis to demonstrate the increased complexity of compositional tasks.

  3. We further clarified World Model usage, data contamination analysis, benchmark results, and the Completion-Focused Reward mechanism.

We believe these additional experiments and clarifications substantially address the reviewers' concerns, particularly the new evidence on cross-model generalizability. We sincerely hope these updates can be taken into consideration in the final assessment.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.