- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focus on the evaluation of multiple user questions within one session. Besides, the authors identify that current evaluation process only introduce additional question based on a fixed process, which may be inconsistent with real world scenarios. Based on this, the authors propose a brand new dataset and analysis current models. Besides, they also give several behavior analysis on the failure pattern.
Critical strengths
- The authors identify that current multi-question evaluation benchmarks ignore that users usually give new prompts when some milestone s appear during the task.
- The authors conduct broad experiments and analysis of their dataset.
Critical weaknesses
The evaluation of multi-question within a single session is valuable. But I think the additional value of this benchmark against existing multi-question benchmark is unclear, which significantly limits the novelty. The authors argue existing benchmarks is inconsistent with real world, but actually, users usually not react directly when some actions are token, cause they cannot monitoring the agent all the time.
Questions for authors
Could the authors clarify the position of this paper? At least I don't think this paper bring something useful to the research community. It looks like a simple and toy experiment, which is only a slight change of current works.
Request to Facilitate Reviewer Discussion
Dear Program Chairs, Senior Area Chairs, and Area Chairs,
Thank you for your time and efforts in coordinating the review process. We are writing to kindly ask for your help in facilitating discussion of our paper.
We submitted our rebuttal several days ago, with detailed responses addressing the reviewers' concerns and important misunderstandings about our work, but have not yet received any replies. To address the reviewers' concerns, we have also revised the manuscript with additional experiments and clarifications of the task formulation, related work, and evaluation protocol.
We would appreciate your help in encouraging the reviewers to engage with our responses and clarify any remaining concerns. We hope to resolve these misunderstandings through discussion before the discussion period ends. Thank you for your time and assistance.
Best regards, Submission252 Authors