- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UserScaler, a synthetic data-generation pipeline for training tool-use agents with user-role participation. UserScaler combines service-grounded environment and persona generation with simulated users that guide trajectory collection, followed by SFT training. This paper shows that agents trained on UserScaler data outperform those trained on synthetic baseline methods without user role guidance on several agent and tool-use benchmarks.
Critical strengths
- This paper studies an important problem: existing synthetic methods often omit the user role information, while UserScaler explicitly incorporates user guidance into the data-generation process.
- This paper used multiple models for training, evaluated on multiple benchmarks, and compared with multiple baselines, which makes it relatively comprehensive. And the reported results show that the proposed method outperforms the baselines in most cases.
- The appendix provides used prompts and hyperparameters, which is useful for reproducing the results.
Critical weaknesses
- The claim is synthesizing environments, however, this paper only uses it for collecting trajectories for SFT training instead of RL in those environments, which undermine the significance of the work. This also causes problem in measuring the quality of the generated environments.
- The main results of Qwen3-8B + UserScaler in Table 2 conflict with it in the appendix Table D.1. The authors need to clarify which results are correct and revise the claims accordingly.
- This paper lacks in-depth analysis of the contribution of each component in UserScaler, especially the persona synthesis, for example, the authors need to provide analysis on diversity, functionality, and more quality metrics of the generated envrionments, tasks, and personas.
- Could the authors provide more details on benchmark evaluation? For example, evaluation user/judge model, the agent budgets, and etc.
Questions for authors
Please see the weakness.
Response to Reviewer gzyX
Thank you for the thoughtful review. We appreciate your recognition of the motivation, experimental coverage, and reproducibility materials. We take your concerns about environment validation, component contributions, and evaluation transparency seriously.
1. Environment generation and the choice of SFT rather than RL.
RL training was not completed within the timeframe of this study. We therefore evaluated whether the generated environments support useful interactive supervision through trajectory collection and SFT. The environments are executable components of collection: agents must invoke their tools, access task-specific data, and change environment state to complete tasks, rather than merely respond to textual descriptions.
We check functionality at two stages. During environment generation, we invoke each service endpoint to verify tool availability. During task generation, we generate the task-specific database or virtual files using the service’s abstract framework and check that the required data are accessible through the environment’s tools. Failed access checks trigger regeneration, with the previous failure information supplied for up to three retries.
Task execution occurs in isolated containers, while the original task data are preserved locally. This enables task-relevant before-and-after state comparison against the original task requirements. Isolation protects the preserved initial state from execution-induced changes. Thus, the validation setup addresses endpoint availability, data accessibility, and observable state changes—not only whether a service starts successfully.
These checks and successful interactive trajectories provide evidence of operational functionality. Nevertheless, we agree that they do not formally guarantee every possible state transition or establish the environments’ effectiveness for RL. We will describe the validation procedure more explicitly and avoid presenting downstream SFT gains as a comprehensive measure of environment quality.
2. Table 2 versus Table D.1.
Thank you for identifying this inconsistency. Table 2 reports the reference results for our main comparisons: Qwen3-8B + UserScaler achieves 44.3% on τ²-bench. The appendix entry was not consistently aligned with the evaluation records used for Table 2. We will correct the affected appendix row, including its per-domain values, and check the remaining appendix tables for consistency. The Table 2 comparisons remain unchanged.
3. Component contributions, particularly persona synthesis.
We agree that the current experiments do not isolate the independent contribution of personas. Personas are integrated into our user-guided collection mode: they specify communication characteristics and domain-knowledge limitations so that the simulated user is not simply a checklist-style task instructor. This describes their intended role, rather than an independently established experimental effect.
The direct versus user-guided ablation changes persona conditioning, multi-turn interaction, and rubric-guided assistance together. Matching trajectory counts does not separate these effects or necessarily match assistant-token supervision. We will therefore interpret it as evidence for the combined collection procedure, not as a controlled demonstration that personas alone cause the improvement.
The paper provides descriptive evidence through service categories, cross-domain compositions, persona archetypes and knowledge levels, task-chain lengths, concrete examples, and the collection-pass-rate analysis. These characterize coverage and collection behavior, but do not comprehensively validate persona realism, behavioral diversity, or semantic correctness. We will distinguish these descriptive statistics from validated quality measurements and narrow the component-level claims. A separate persona ablation is not available in the current study.
4. Benchmark evaluation details.
Both the evaluation user simulator and the judge use Qwen3.5-397B. All Table 2 checkpoints are evaluated locally under the same benchmark setup. We retain each benchmark’s default agent-budget settings, including its applicable interaction and tool-use limits, without additional modifications. Apart from the common user-simulator and judge models, we follow the benchmark defaults.
We will make these settings explicit, including the role of each model and the benchmark-default budget configurations. We will also distinguish the evaluation user simulator from the training-data collection user: privileged rubric guidance is a collection mechanism, while evaluation follows the benchmark’s user-simulator configuration.
Thank you again for highlighting these gaps. We will use the revision to clarify the implementation and ensure that our conclusions reflect the scope of the existing evidence.