acceptodds
Withdrawn from ICLR 2027

UserScaler: Beyond Environment Scaling—Unlocking the User Dimension for Agent Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling the training of large language model (LLM) agents relies on diverse and reliable environments. Existing methods leverage automatic pipelines to synthe- size interactive, stateful, and scalable environments. Nevertheless, they overlook the substantial gap between idealized users and real-world users: realistic users often hold partial intents, exhibit variable phrasing styles, and react dynamically as interactions proceed. This discrepancy hinders the effectiveness of existing agents in real-world application. To tackle this issue, we propose UserScaler, which jointly generates plausible users and environments to bridge the gap be- tween agent training and real-world deployment. UserScaler consists of three core components: (1) Realistic user-centric environment synthesis: a search agent constructs each environment following specifications of real services and derives user personas from market research of the target service; (2) Realistic user-driven task generation: user requests are synthesized conditioned on user personas and sampled toolchains; (3) User-guided trajectory collection: during rollout, simu- lated users actively engage in the interaction, whose behaviors are guided by an automatically derived rubric covering both process and outcome. We instanti- ate 209 environments (109 single-service and 100 cross-domain compositions), each equipped with roughly 50 user personas, yielding 12,073 training trajecto- ries. Training Qwen3 models from 1.7B to 8B on this data consistently improves pass rates over corresponding baselines trained with EnvFactory, AWM, and En- vScaler across three agentic benchmarks: the 8B model reaches 44.3% on τ2 -bench, 19.8% on VitaBench, and 11.3 on UserBench. Under a matched budget of 6,474 trajectories, user-guided rollouts outperform direct ones, confirming that structured user modeling provides signal beyond additional data volume. We re- lease the environments, training data, and models.

Primary area:
datasets and benchmarks
Keywords:
User-Centric Environments, LLM Agents, Long-Horizon Task Solving, Implicit Intent Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gzyXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UserScaler, a synthetic data-generation pipeline for training tool-use agents with user-role participation. UserScaler combines service-grounded environment and persona generation with simulated users that guide trajectory collection, followed by SFT training. This paper shows that agents trained on UserScaler data outperform those trained on synthetic baseline methods without user role guidance on several agent and tool-use benchmarks.

Critical strengths
  1. This paper studies an important problem: existing synthetic methods often omit the user role information, while UserScaler explicitly incorporates user guidance into the data-generation process.
  2. This paper used multiple models for training, evaluated on multiple benchmarks, and compared with multiple baselines, which makes it relatively comprehensive. And the reported results show that the proposed method outperforms the baselines in most cases.
  3. The appendix provides used prompts and hyperparameters, which is useful for reproducing the results.
Critical weaknesses
  1. The claim is synthesizing environments, however, this paper only uses it for collecting trajectories for SFT training instead of RL in those environments, which undermine the significance of the work. This also causes problem in measuring the quality of the generated environments.
  2. The main results of Qwen3-8B + UserScaler in Table 2 conflict with it in the appendix Table D.1. The authors need to clarify which results are correct and revise the claims accordingly.
  3. This paper lacks in-depth analysis of the contribution of each component in UserScaler, especially the persona synthesis, for example, the authors need to provide analysis on diversity, functionality, and more quality metrics of the generated envrionments, tasks, and personas.
  4. Could the authors provide more details on benchmark evaluation? For example, evaluation user/judge model, the agent budgets, and etc.
Questions for authors

Please see the weakness.

1 comment

Response to Reviewer gzyX

Thank you for the thoughtful review. We appreciate your recognition of the motivation, experimental coverage, and reproducibility materials. We take your concerns about environment validation, component contributions, and evaluation transparency seriously.

1. Environment generation and the choice of SFT rather than RL.

RL training was not completed within the timeframe of this study. We therefore evaluated whether the generated environments support useful interactive supervision through trajectory collection and SFT. The environments are executable components of collection: agents must invoke their tools, access task-specific data, and change environment state to complete tasks, rather than merely respond to textual descriptions.

We check functionality at two stages. During environment generation, we invoke each service endpoint to verify tool availability. During task generation, we generate the task-specific database or virtual files using the service’s abstract framework and check that the required data are accessible through the environment’s tools. Failed access checks trigger regeneration, with the previous failure information supplied for up to three retries.

Task execution occurs in isolated containers, while the original task data are preserved locally. This enables task-relevant before-and-after state comparison against the original task requirements. Isolation protects the preserved initial state from execution-induced changes. Thus, the validation setup addresses endpoint availability, data accessibility, and observable state changes—not only whether a service starts successfully.

These checks and successful interactive trajectories provide evidence of operational functionality. Nevertheless, we agree that they do not formally guarantee every possible state transition or establish the environments’ effectiveness for RL. We will describe the validation procedure more explicitly and avoid presenting downstream SFT gains as a comprehensive measure of environment quality.

2. Table 2 versus Table D.1.

Thank you for identifying this inconsistency. Table 2 reports the reference results for our main comparisons: Qwen3-8B + UserScaler achieves 44.3% on τ²-bench. The appendix entry was not consistently aligned with the evaluation records used for Table 2. We will correct the affected appendix row, including its per-domain values, and check the remaining appendix tables for consistency. The Table 2 comparisons remain unchanged.

3. Component contributions, particularly persona synthesis.

We agree that the current experiments do not isolate the independent contribution of personas. Personas are integrated into our user-guided collection mode: they specify communication characteristics and domain-knowledge limitations so that the simulated user is not simply a checklist-style task instructor. This describes their intended role, rather than an independently established experimental effect.

The direct versus user-guided ablation changes persona conditioning, multi-turn interaction, and rubric-guided assistance together. Matching trajectory counts does not separate these effects or necessarily match assistant-token supervision. We will therefore interpret it as evidence for the combined collection procedure, not as a controlled demonstration that personas alone cause the improvement.

The paper provides descriptive evidence through service categories, cross-domain compositions, persona archetypes and knowledge levels, task-chain lengths, concrete examples, and the collection-pass-rate analysis. These characterize coverage and collection behavior, but do not comprehensively validate persona realism, behavioral diversity, or semantic correctness. We will distinguish these descriptive statistics from validated quality measurements and narrow the component-level claims. A separate persona ablation is not available in the current study.

4. Benchmark evaluation details.

Both the evaluation user simulator and the judge use Qwen3.5-397B. All Table 2 checkpoints are evaluated locally under the same benchmark setup. We retain each benchmark’s default agent-budget settings, including its applicable interaction and tool-use limits, without additional modifications. Apart from the common user-simulator and judge models, we follow the benchmark defaults.

We will make these settings explicit, including the role of each model and the benchmark-default budget configurations. We will also distinguish the evaluation user simulator from the training-data collection user: privileged rubric guidance is a collection mechanism, while evaluation follows the benchmark’s user-simulator configuration.

Thank you again for highlighting these gaps. We will use the revision to clarify the implementation and ensure that our conclusions reflect the scope of the existing evidence.

Reviewer_Za6kOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a persona-based data synthesis pipeline termed UserScaler, which operates by first gathering real service specs and market research data to generate mock service environments and user personas. Then the generated personas are used to guide task synthesis and multi-turn trajectory collection, where simulated users receive privileged rubric information to assist the agent when it stalls.
The authors train Qwen3 models on the collected SFT trajectories, and evaluate on multiple tool-call benchmarks with simulated users.

Critical strengths
  1. Modelling realistic and diverse user personas is an important direction for realistic data synthesis and agent training improvement.
  2. This paper provides detailed breakdown on the coverage of the synthesized environment, concrete examples of the synthesized tasks and grading output, which are useful for the audience to understand the mechanism.
Critical weaknesses
  1. In the experiments, the authors claim that all generation uses Qwen3.5-397B as the backbone LLM. However, it is unclear if the authors use the original open-sourced data from other baselines ("All methods apply SFT to the same Qwen3 base models using their respective synthesized data") or existing checkpoints (The authors explain that AWM's 1.7b reults are missing because AWM does not release a 1.7B checkpoint). Were the baseline data reproduced using the same Qwen3.5-397B backbone for trajectory generation and other parts of their pipelines? If not, then the differences in trajectory-generation models, data budgets, and training configurations may confound the main results in Table 2, and the improvements therefore could not be solely attributed to UserScaler's methodology. The lack of clarity regarding baseline reproduction and experimental controls significantly weaken the fairness of the comparison.
  2. Cost is an important aspect of data synthesis methodology, yet the authors do not provide an explicit cost analysis or statistics of the proposed UserScaler method (e.g., total token consumption, monetary cost, or wall-clock time to generate each environment). Moreover, this paper does not provide a qualitative nor quantitative cost comparison between UserScaler and the other baseline methods -- only surface-level statistics such as the number of environments is provided, in which makes it difficult to assess the cost-effectiveness of the proposed pipeline.
  3. The authors use multiple types of checks to ensure correctness of a solution. However, these types of checks (i.e., tool-use checks, artifact checks, execution checks, and LLM-based checks) do not necessarily guarantee the correctness of the final environment state. For instance, the example in Figure K.1, the flight-booking task receives a passing score without an explicit programmatic check that ensures the final reservation state matches with the desired one.
Questions for authors
  1. The Qwen3-8B τ2\tau^2-bench and VitaBench results are different from the ones in Table D.1 and Table D.2 (i.e., 44.3 average in Table 2 vs. 48.9 average in D.1; 19.8 average in Table 2 vs. 19.3 average in D.2). Can you explain these inconsistencies?
  2. For 100 cross-domain scenarios construction, how exactly do you select 2-3 validated services? (i.e., how the model selects and composes 2-3 services, how do you ensure that the selected services are compatible in terms of business logic?)
  3. L427-L429 indicates there are 106 + 94 = 200 environments, which is inconsistent with the reported 209 environments.
  4. Figure 2 and Figure 3 are too small to read and are visually crowded. Also in Figure 2, stage 2 step 4 -- two checkboxes are missing for "Execution checks" and "File checks". In stage 3 step 3, the checkmarks are weirdly positioned; the "document" icon in stage 3 step 4 appears to be cropped at the bottom. There are more issues similar to this in the figures as well. Can you fix them?
  5. Can you provide more implementation details for the search agent and code agent? What harness is used, how information is actually retrieved, and are they using the same Qwen3.5-397B LLM backbone?
1 comment

Response to Reviewer Za6k — Questions for Authors

Thank you for these specific questions. We address them in order.

1. Inconsistencies between Table 2 and Tables D.1–D.2.

Table 2 reports the reference results for our main comparisons: UserScaler-8B achieves 44.3% on τ²-bench and 19.8% on VitaBench, both measured as Pass@1. The appendix entries were not consistently aligned with the evaluation records used for Table 2. We will correct the affected rows in Tables D.1–D.2, including their per-domain values, so that the main text and appendix report the same results. The comparisons in Table 2 remain unchanged.

2. Cross-domain scenario construction.

The model selects groups of two or three services from the validated single-service pool. It is instructed to propose realistic, coherent combinations and provide an explicit rationale describing the business workflow, the role of each constituent service, and why those services are appropriate to combine. Selection is therefore guided by a plausible business use case rather than arbitrary grouping. For example, an airline service, a calendar, and an itinerary service can support booking a flight, scheduling the trip, and organizing the itinerary. One service is designated as the primary environment to anchor the user persona. The required rationale makes the intended business relationships explicit; it is a plausibility constraint rather than a formal guarantee of end-to-end semantic compatibility. We will clarify this procedure in §3.1.1.

3. Environment counts.

The full generated pool contains 209 environments: 109 single-service environments and 100 cross-domain compositions. For the ablation in §4.4, we selected 5,024 training trajectories from each environment type to match trajectory counts. The resulting subsets cover 106 single-service environments and 94 cross-domain compositions, respectively. The remaining nine environments are simply not represented in these sampled ablation datasets; they were not excluded from the full generated pool. We will revise §4.4 to distinguish the environment coverage of the matched-size ablation subsets from the total generated environment count.

4. Figures 2 and 3.

We will improve readability by enlarging labels and simplifying the layouts. In Figure 2, we will restore the missing “Execution checks” and “File checks” checkboxes, correct the checkmark alignment, and fix the cropped document icon. We will inspect both figures at the final manuscript size to ensure that their labels and graphical elements are legible.

5. Search and code agent implementation.

We use claude code as the agent harness, with Qwen3.5-397B as the LLM backbone for both the search and code agents. The harness, model backbone, and retrieval tool serve distinct roles. Retrieval begins with a supplied topic, such as “project management,” as described in §3.1.1 and Appendix B. The search agent calls the Brave Search API to obtain relevant information about real-service specifications, including endpoint semantics, field schemas, and domain conventions. For persona generation, it retrieves information from industry surveys and market research. The supplied topic guides retrieval, while the retrieved evidence informs service planning and code generation. The code agent implements the mock service and fixture data; validation failures are fed back with error information for up to three repair iterations. We will expand Appendix B to state the harness, retrieval API, and shared model backbone explicitly.

Reviewer_fzUSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

UserScaler generates training data for tool-using agents by pairing synthetic services with user profiles. A simulated user reveals requirements over several turns and can use the task rubric to help when the agent gets stuck. The authors report 12,073 retained trajectories and use them to fine-tune Qwen3 models from 1.7B to 8B. The largest reported gains over prior methods are on VitaBench.

Critical strengths

The paper addresses a practical problem: users often leave out details that an agent needs before it can act. The pipeline can generate examples of agents asking for those details and then carrying out the task in an executable environment.

The VitaBench result is promising. At the same trajectory count, the 8B model trained on user-guided data scores 15.6%, compared with 6.3% for direct data. The full dataset reaches 19.8% in Table 2.

Critical weaknesses
  1. The main ablation compares giving the agent all requirements upfront with a multi-turn user who has a persona and can give rubric-based hints. This leaves it unclear whether the gain comes from the persona or hints, or simply from training on multi-turn dialogue. APIGen-MT and PersonaForge already use simulated users who reveal requirements over several turns. A comparison with an ordinary multi-turn simulator on the same tasks would help establish what UserScaler adds.

  2. The results and baseline settings need clarification. For the 8B UserScaler model, Table 2 reports 44.3% on τ²-bench, while Table D.1 reports 48.9%. Some baseline scores also differ considerably from the original papers: EnvFactory-1.7B scores 12.2% in Table 2 versus 15.57% for the SFT model in EnvFactory’s Table 1. It is unclear whether the authors retrained the baselines or reevaluated released checkpoints, especially for AWM, whose original results use RL. The data counts also need clarification: if both disjoint 6,474-trajectory subsets came from the stated mixed corpus, they would require at least 12,948 trajectories, exceeding the reported 12,073.

Questions for authors
  1. How were the baseline results in Table 2 obtained: taken from the original papers, reproduced by evaluating released checkpoints, or obtained by training new models? Several scores differ from the published results. For example, Table 2 reports 12.2% for EnvFactory-1.7B on τ²-bench, compared with 15.57% for the SFT model in EnvFactory’s Table 1. What accounts for the lower score?

  2. Section 4.1 says all baselines use SFT, but AWM’s original results are for RL-trained models. Did you run SFT on AWM-generated trajectories or evaluate the released RL checkpoint? If you retrained the baselines, what training data and token budgets did you use? Were all models evaluated on the same tasks with the same user simulator and scoring rules?

  3. Where did the two 6,474-trajectory subsets come from? Please give the counts by collection mode and filtering stage.

  4. On the same tasks, how does an ordinary multi-turn user compare with a grounded persona with and without rubric hints? Please match the amount of assistant supervision and report how often the agent actively asks for missing preferences.

1 comment

Response to Reviewer fzUS

Thank you for the careful review and for recognizing the motivation of UserScaler and its VitaBench results. We address the concerns below.

Ablation and attribution. We agree that user-direct versus user-partial changes several factors together: single- versus multi-turn interaction, persona conditioning, and rubric-guided hints. Matching 6,474 training trajectories per condition controls trajectory count, but not necessarily assistant-token count. We will narrow our conclusion accordingly: Table 3 shows that our combined user-guided collection procedure outperforms direct collection at matched trajectory count on VitaBench (15.6% versus 6.3%). It does not establish the independent contribution of personas or hints. We will remove language making that stronger attribution and clarify the distinction from prior multi-turn simulators, including APIGen-MT and PersonaForge. An ordinary multi-turn comparator and component-wise ablations would be needed to isolate those effects; we do not have those results. We also have not separately measured how often trained agents ask for missing preferences.

Baseline provenance and evaluation. The baseline scores in Table 2 were neither copied from the original papers nor obtained by retraining their methods. We downloaded the publicly released EnvFactory, AWM, and EnvScaler checkpoints, deployed them locally, and evaluated them alongside our SFT-trained UserScaler checkpoints. All checkpoints in Table 2 were evaluated under the same local benchmark setup, with Qwen3.5-397B used as the judge model. Apart from this common judge, we made no other changes to the benchmark evaluation setup.

We will correct the statement in §4.1 that “all methods apply SFT”: we SFT-trained UserScaler but evaluated the baselines’ released checkpoints without retraining them. In particular, we did not SFT-retrain AWM or replace its original RL training. We therefore do not claim matched training data or token budgets across methods. EnvFactory-1.7B’s 12.2% in Table 2 is our local evaluation of its released checkpoint, whereas 15.57% is the result reported in the EnvFactory paper. The shared local setup supports comparisons within Table 2; we cannot identify the specific cause of the difference from the published score without isolating differences between the two evaluations.

Table 2 versus Table D.1. Thank you for catching this inconsistency. Table 2 contains the result used for our main comparison: UserScaler-8B scores 44.3% overall on τ²-bench (46.0% airline, 61.9% retail, 25.0% telecom). The corresponding Table D.1 row is incorrect; it does not represent a different checkpoint or evaluation setting. We will correct the entire row and check the remaining appendix entries. The Table 2 comparisons are unchanged. Trajectory accounting. Both collection modes used the same pool of 18,525 task configurations. After quality filtering, direct mode retained 6,474 trajectories, while user-guided mode retained 12,073 trajectories. The full UserScaler model was trained on the 12,073 retained user-guided trajectories. For the matched-size ablation, we trained user-direct on the 6,474 retained direct-mode trajectories and user-partial on a 6,474-trajectory subset of the retained user-guided trajectories. Thus, the two ablation sets consist of distinct trajectories generated in different modes; they are not two disjoint subsets of the 12,073 user-guided trajectories. Their combined size need not be bounded by 12,073. We will clarify the mode-wise counts and selection procedure in §4.3 and the Table 3 caption.

We will correct these reporting issues and limit our conclusions to what the existing experiments establish.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.