- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces StressSim, which is a simulated user searching for challenging conversations while preserving original task. The optimization procedure evolves planner prompts separately for each task, using task-completion scores and automated conversation-validity checks. The experiments on MultiWOZ and τ-bench’s retail and airline domains show substantial reductions in agents’ repeated-success reliability, supported by a human validation study.
Critical strengths
- It introduces a different but important evaluation problem that evaluating failure modes that caused by changing or obstructing the task.
- It is a practical approach that the pipeline consists of goal tracking, planning, and utterance generation that we can exam each components easily and the provided prompts and conversation examples help readers inspect how failures arise.
- The authors assess their validity judge using 100 conversations labelled by two humans, examine transfer across target models, and provide concrete agent-failure examples. These are valuable checks beyond aggregated scores.
Critical weaknesses
-
The ablation study about the beta sensitivity trade-off is questionable to me. Formula R_adv shows that beta greater than 1 already puts all valid candidates ahead of invalid ones. Increasing beta from 2 to 5 or 10 would therefore leave the candidate ranking unchanged. Could the authors clarify how beta actually affects the search, or whether the implementation includes an additional mechanism?
-
StressSim receives 20 adaptive optimization rollouts per task, while the baselines use fixed instructions. A budget-matched random prompt search or best-of-20 strategy baseline is needed.
-
The evidence for intent preservation could be stronger. Human audit for StressSim conversations that pass the validity judge and cause the agent to fail.
Questions for authors
See weakness.
2 comments
Thanks for your clarification. After reading the rebuttal, I would maintain my original scores.
We thank the reviewer for the detailed review feedback and for recognising the importance of the evaluation problem and the practicality of the modular pipeline. Please find below our responses to the concerns raised:
W1. We thank the reviewer for pointing out this discrepancy. The observation is correct, and led us to find two description errors as explained in GR4. The differences across Fig. 3a are of the order of two tasks out of 50 from single optimisation runs, and we withdraw the trade-off interpretation: the optimisation is insensitive to beta in the tested range. We have corrected Section 3, Algorithm 1 and revised the interpretation in Section 4.
W2. Budget-matched baselines: We have run the suggested experiments to compare performance against an equal budget random search (best of 20 seed-derived candidates, one rollout each) and the unoptimized seed planner. The results are summarised in GR1. The seed planner prompt applies no adversarial pressure and has poor validity, while the random search matches StressSim on MultiWOZ but at IGA 0.84, and is weaker on every metric on both tau-bench datasets. StressSim under the same budget reaches much lower pass^4 at IGA 0.92-1.00. This illustrates the impact of the reflective, genetic optimisation based approach in terms of applying adversarial pressure while complying with validity constraints.
W3. Human audit of accepted, failure-inducing conversations: We have conducted the suggested additional human validation of 100 StressSim trajectories that were marked valid by the judge, yet led to task failures, sampled across the nine agent-benchmark pairs in Table 1 (details in GR2). Two annotators followed the same protocol as the original study. Annotator 1 found 84 of the 100 conversations valid and Annotator 2 found 92 valid, with 92% inter-annotator agreement (Cohen's kappa 0.63). This indicates that the failures StressSim exposes are predominantly caused by legitimate user behaviour rather than by trajectories that exploit the judge.