- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces AgentMercury, a framework that first builds executable environments from business scenarios and then generates tasks from those environments. The authors develop PLANET to construct worlds with multiple services, tools, persistent state, and cross-service constraints. They generate 4,783 environments and use the resulting tasks to train Qwen3.5 models with RL, reporting improvements on enterprise workflows and several out-of-domain benchmarks.
Critical strengths
I think the most interesting part of this paper is its world-first approach. Many existing methods build environments around specific tasks, whereas AgentMercury starts with a persistent business world where different services interact and share state, and then generates tasks from it. This makes the environment less tied to any single predefined task and allows the same world to support different kinds of interactions. I see this as a worthwhile direction, especially for agents that need to handle long workflows across multiple systems.
Critical weaknesses
My main concern is that the paper does not really establish the value of its world-first design. The experiments show that RL on the generated tasks improves model performance, but they don't tell us whether the improvement comes from the world-first approach itself. Would a task-first or another environment generation method achieve similar results under the same training budget? I think this is the most important question the paper needs to address. The framework also combines world reuse, multi-service structure, persistent state, and cross-service constraints, without ablations that separate their effects. Even with the positive results, it is hard to tell which of these design choices actually matter.
Some experimental results also raise questions. On τ³ Telecom, the 35B base model scores 49.1 ± 49.8, while the 4B base model reaches 92.5. Such large variance makes me less convinced by the authors' claims about policy stability. The 4B SAO results are also omitted due to unstable training, which I think is worth reporting.
Finally, the environment-authoring results improve from 3.3% to 83.3%, but the evaluation uses only 30 held-out scenarios. The success rate also drops to 10% with recipe conditioning. The authors attribute this to the model having internalized the construction process, but it could just as well reflect sensitivity to the input format. The current experiments do not distinguish between these explanations.
Questions for authors
-
Could the authors compare AgentMercury with a task-first or other environment generation approach under the same RL training budget? I think this comparison is important for understanding whether the gains actually come from the world-first design. Ablations on world reuse, persistent state, and cross-service constraints would also help clarify which parts of the framework matter most.
-
Could the authors explain the unusually large variance on τ³ Telecom, particularly for the 35B base model? I would also like to see the omitted 4B SAO results, even if training was unstable.
-
What evidence supports the authors' explanation for the drop from 83.3% to 10% with recipe conditioning? Could this simply reflect sensitivity to the input format rather than interference with an internalized construction process?
6 comments
W3 and Q3. Authoring sample size and why the recipe hurts.
Finally, the environment-authoring results improve from 3.3% to 83.3%, but the evaluation uses only 30 held-out scenarios. The success rate also drops to 10% with recipe conditioning. The authors attribute this to the model having internalized the construction process, but it could just as well reflect sensitivity to the input format. The current experiments do not distinguish between these explanations.
| Qwen3.5-35B-A3B authoring | Brief only | Brief + recipe | Upto 200 instances (recipe) |
|---|---|---|---|
| Before construction-trace fine-tuning | 3.3% | 20.0% | 77.5% |
| Digest only | 10.0% | 76.7% | 67.0% |
| Exemplar only | 16.7% | 73.3% | 71.0% |
| Digest + Exemplar | 20.0% | 10.0% | 25.5% |
| After fine-tuning | 83.3% | 10.0% | 65.5% |
The recipe changes success in opposite directions before and after fine-tuning. Each condition evaluates 30 held-out briefs, with success requiring all 12 structural validators (Table 3; Appendix F). The paper reports that 27 fine-tuned recipe-conditioned outputs fail the cross-service check. This identifies an unmet structural requirement in the outputs, without identifying how it arose. The recipe jointly adds an invariant digest and an exemplar, changing both content and input format. Format sensitivity, exemplar content, effective generation headroom, and interference with a learned procedure are not separated by this comparison.
We have therefore withdrawn interference with an internalized construction process as an established explanation and retained it only as a hypothesis. The revised authoring claim is limited to the tested brief-only interface. The recipe-conditioned failure is established; its cause and generality remain unresolved. Distinguishing these explanations would require paired tests of the original checkpoint with digest-only, exemplar-only, and combined inputs under equal effective output budgets.
Dear Reviewer,
We sincerely appreciate your time and thoughtful feedback on our work.
We understand that this is a particularly busy period and that reviewing additional materials can be demanding. Nevertheless, we would be very grateful if you could take another look at our responses and revised manuscript, where we have made substantial efforts to address your concerns.
We hope these revisions help clarify our contributions and resolve the issues you raised. If you find that our responses and revisions adequately address your concerns, we would sincerely appreciate your reconsideration of the overall assessment and score.
Thank you again for your valuable time and consideration.
Why construct the world first, and what the current evidence establishes
Thank you for recognizing the world-first direction and for distinguishing the utility of the generated training data from evidence about individual design choices.
W1 and Q1. World-first design, matched-budget alternatives, and component ablations.
We have not completed a matched-RL-budget comparison with another environment source or component-removal ablations. The reported improvements establish the training utility of the complete pipeline; they do not show that world-first construction causes those improvements or outperforms a task-first source. The Introduction and Section 4.1 now make that distinction explicit.
The reason for constructing a world first is to give task synthesis a reusable set of business entities, ownership relations, and operations. The composer selects owned records in that world and attaches situation-specific evidence, instructions, and grading criteria. Appendix A illustrates this with two released portfolio investigations that reuse the same customer record and tool interface. One attaches incident-and-coverage evidence; the other attaches quality-issue evidence. Their complete investigation structure is shared after entity/value rebinding. This is evidence of reuse across situations, not of distinct whole-task objective families or superior training efficiency.
The components serve different purposes: persistent state retains action consequences within an episode; services expose records and communication channels; cross-service constraints specify consistency requirements. These implemented roles explain the design, but do not establish each component's contribution to learning. We also distinguish declaring a constraint from enforcing it during every transition.
Appendix A provides two supporting analyses within that scope. Its construction-cost identity explains how compatible tasks can share world-building overhead; AWM and Agent-World also reuse environments, so this is not an asymptotic or measured cost advantage over them. Its scripted execution example verifies one request-grounded state condition using native tools, without establishing full-task success or an RL effect. The requested comparative learning evidence remains a substantive limitation.
W2 and Q2. Telecom variance and the omitted 4B SAO results.
We agree that the unusually large variance on τ³ Telecom warrants further discussion. Table 2 reports 49.1 ± 49.8 for the 35B base model over three evaluation runs, compared with 92.5 ± 1.0 for the 4B base model. Such variability is not necessarily unique to our setting, as instability and sensitivity in base-model performance have also been discussed in prior studies and external reports. Nevertheless, without the original run-level outputs and serving configurations, we cannot determine the specific cause of the observed variance.
Interestingly, both RL training methods show a trend toward reduced variability relative to the 35B baseline, with the standard deviation decreasing from 49.8 to 20.5 (GRPO) and 23.8 (SAO). While these results suggest that RL training may help mitigate some of the underlying variability, the remaining variance is still substantial, and three evaluation runs are insufficient to establish a general stabilization effect. We have therefore revised Section 4.3 and the Conclusion to highlight this observed trend without making a strong causal claim about policy stability.
We also agree that the excluded 4B SAO outcome should be reported. Unfortunately, validated benchmark values are not available in the recovered materials, so we cannot provide reliable numerical results or infer them from the training curve. Appendix E.3 and the Conclusion now explicitly acknowledge the exclusion and the reported training instability. Appendix C also clarifies that the optimizer comparison uses broadly comparable infrastructure but does not isolate the optimizer's causal effect.