- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds a pipeline that turns public OpenAPI specifications into isolated, stateful MCP environments, each backed by a relational database that enforces application rules. On top of these environments, it generates multi-step tasks from executed reference programs, and each task can be posed with either a scripted or a simulated user. The paper also introduces an evaluation suite and uses the resource for mixed RL training of a small model, reporting gains on BFCL, a three-domain τ benchmark and the suite.
Critical strengths
Grounding tasks in executed reference programs is the right design for RL on stateful tools. Because each request is written from an actual execution and its database changes, every task has a checkable target state, and the reward can rely mainly on state checks rather than only on model judgment.
Critical weaknesses
- The contribution is not separated from the closest prior work. Public work already synthesizes stateful, database-backed tool environments from API or tool specifications and trains agents on them with RL: AgentScaler (arXiv 2509.13311), EnvScaler (arXiv 2601.05808) and AWM (arXiv 2602.10090). AWM in particular also exposes database-backed environments as MCP tools, trains small Qwen models with GRPO, and evaluates on BFCL and τ²-bench. The paper neither discusses nor compares against any of these. Its own distinctive choices, starting from real OpenAPI specifications and posing each task in two user formats, are never shown to matter, so the contribution cannot be distinguished from existing pipelines.
- The experiments do not show that these environments are good training material. The before/after comparison favours the trained model: the base score is the worst of several runs while the trained model is run once, and on the paper's own suite the same model acts as user simulator, training reward judge and evaluation judge. More importantly, the paper compares only against no training. Without a comparison to training on other environments (e.g., AWM's), even the reported gains do not show that these environments are better than what already exists.
Questions for authors
- How does the pipeline differ from AgentScaler, EnvScaler and AWM, both in how environments are built and in the environments it produces? At a matched task budget, does training on Echoverse-MCP outperform training on their environments?
- Are the 20 environments in the evaluation suite disjoint from the 45 environments used to generate training tasks?