- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed a new agent benchmark DynaWorld, where agents interact with user simulators under evolving conditions. To adept the evolving environment, the paper proposed DynEvo which combines multi-agent collaboration with online self evolution through a tool workbench. After each episode, the service agent reflects on the trajectory and episode-level feedback update notes and skills, while strategies are reviewed periodically. The experiments shows DynEvo achieves consistent gains across dynamic scenarios and outperforms existing evolving agents.
Critical strengths
S1. DynaWorld is the first interactive service benchmark where the environment itself changes across episodes. The event design (policy, user population, objective, time-limited rules) and the fixed event schedules make it possible to measure, in a repeatable way, how an agent’s performance drops and recovers when conditions change.
S2. The analysis by event type (Figures 4–5) gives results that static benchmarks cannot: changes in user population cause the largest and slowest-recovering drop, while time-limited changes are absorbed quickly. These findings are useful on their own, regardless of how DynEvo compares to other methods.
Critical weaknesses
W1. The claim that DynEvo beats “strong self-evolving baselines” is not supported. None of the four baselines keeps and reuses experience across episodes, while the methods that do, and that the paper itself cites (ACE, Memento, ExpeL, Mem0), are not compared. At least one such method should be run with the same base model and agent setup.
W2. The judge-based Overall Score is used both to evaluate all methods and to guide DynEvo’s updates between episodes. This gives DynEvo a feedback signal the baselines do not get, and it risks DynEvo learning to please the judge rather than to serve users better. No confidence intervals are reported, although they could be computed from the existing episodes without new runs.
Questions for authors
Is the judge (and the user simulator) also GPT-5.2? EvoAgentX is said to use episode-level feedback. Is that the same Overall Score DynEvo uses, and do the other three baselines get anything between episodes? How many runs are behind Table 3? Temperature alone moves the score by about a point (Table 11), and the main gap is 4.5.
3 comments
The author's rebuttal has fully addressed my concerns. The advantage over memory-based baselines is solid especially on Res. Rate. However, the authors admitted to use GPT-5.2 with temperature = 1.0 and top-p = 1.0 for both the User Simulator and the Judge Agent implies reasoning_effort is none. I would doubt it may not be an ideal setting for Judge Agent. Therefore, I keep my rating.
We sincerely thank the reviewer for the insightful and constructive feedback! Your comments have helped us improve the paper. During the rebuttal period, we conducted human validation with three trained annotators to further assess the reliability of the User Simulator and Judge Agent under the current settings. The protocol and results have been added to Appendix C.1 (Experimental Setup).
(a) Judge Agent. The annotators independently assessed the trajectories using the same evaluation criteria as the Judge Agent. The Spearman correlation between the averaged human Overall Scores and the Judge Agent’s Overall Scores is 0.814, indicating strong alignment with human judgments.
(b) User Simulator. The annotators reviewed the user profiles and trajectories for persona fidelity, goal consistency, factual consistency, and response and termination behavior. With approval from at least two annotators required to pass, the majority-vote pass rates are 99.29%, 100%, 100%, and 97.86%, respectively, supporting the User Simulator’s consistency with the specified profiles and objectives and the appropriateness of its conversational behavior.
We appreciate your feedback on the Judge Agent’s reasoning configuration and will take this suggestion into consideration in future work. Thank you again for your time and valuable feedback throughout the review process!
Author Response to Reviewer CWWJ
Dear Reviewer CWWJ:
We sincerely thank you for the insightful and constructive feedback! We appreciate your recognition of DynaWorld as the first interactive service benchmark where the environment itself changes across episodes, its reproducible event design, and the event-level insights that static benchmarks cannot provide. We address the concerns and clarify the experimental protocol below.
W1: Comparison with Experience-Learning Methods.
We thank the reviewer for this valuable suggestion. Among the existing baselines, AutoAgents and EvoAgentX both include memory and experience-reuse components (Appendix C.1, Table 10), and we have clarified these mechanisms in the revised manuscript. To compare with methods specifically designed for memory-based experience learning, we additionally evaluate ExpeL (AAAI 2024)[1] and ACE (ICLR 2026)[2] using the same GPT-5.2 backbone and settings as DynEvo, with results shown below.
Dynamic environments require both experience accumulation and timely strategy adjustment when conditions change. Memory-based methods emphasize updating and reusing experience and rules, but such reuse may not readily translate into changes in overall handling principles. DynEvo combines experience management across different timescales with explicit Strategy Reconstruction, supporting longer-term adaptation while preserving useful experience, and achieves higher Overall Score and Res. Rate in these comparisons. We have added these results to Table 3 and clarified the baseline settings in Appendix C.1 (Baselines) of the revised manuscript.
W2 & Q2: Fairness of Scoring Feedback and Evaluation Metrics.
We thank the reviewer for pointing out this ambiguity. EvoAgentX and the other self-evolving baselines receive the same per-episode Overall Score feedback as DynEvo to guide their respective updates. Thus, DynEvo has no additional advantage in access to this feedback. We have clarified this shared feedback protocol in Appendix C.1 (Baselines) of the revised manuscript.
This design draws on feedback mechanisms in real-world dynamic interactive service environments, where user feedback is commonly used both to evaluate service quality and to guide future service improvement. We also report Res. Rate, which is determined by the User Simulator based on task completion, does not rely on the Judge’s scores, and is not used for evolution updates. DynEvo achieves the highest average Overall Score and Res. Rate across the eight scenarios, demonstrating advantages in both service quality and task completion.
Q1: Model Settings of the Judge Agent and User Simulator.
We thank the reviewer for pointing out this ambiguity. Drawing on the evaluation design of prior agent benchmarks such as τ-bench[3], we use GPT-5.2 with temperature = 1.0 and top-p = 1.0 for both the User Simulator and the Judge Agent. We have explicitly reported these settings in Appendix C.1 (Experimental Setup) of the revised manuscript.
W2 & Q3: Number of Runs and Confidence Intervals.
We thank the reviewer for raising the question of statistical reliability. Following τ-bench[3], each method in our main experiments was evaluated over three independent runs. We report 95% Student’s t confidence intervals across runs for the average Overall Score and Res. Rate, as shown below. We have clarified the number of runs in Appendix C.1 (Experimental Setup) and added the confidence intervals and statistical protocol to Appendix C.3.1 (Robustness) of the revised manuscript.