- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether LLMs can tell what has been said in a multi-turn dialogue apart from what actually remains in effect. It introduces Intent-Eval, a benchmark across Actions, Code, Database and Math, where the same task is run under different conditions: no interruption, a neutral clarification, a proposed change that is rejected, and the same change accepted. Matched single-turn versions separate multi-turn effects from task difficulty, and longer dialogues test whether the interference grows or persists. The authors find that models lose accuracy even from neutral clarifications, and more so when a change is proposed, whether it is rejected or accepted. Their errors often contain content that was rejected or replaced. To address this, they propose Intent-OPSD, a Teacher-Student self-distillation method where the Teacher sees the single-turn task matching the user’s final decision and the Student learns from the full dialogue. It improves over the base models and over standard fine-tuning.
Critical strengths
[1] The paper takes a focused look at a problem that earlier work only touches on. Prior work such as “LLMs get lost in evolving user intent” treats argument revision as one setting among several. Here, comparing clarification, rejection and acceptance on the same tasks, with matched single-turn controls, isolates whether a mentioned change derails the task that should remain in effect. This matters because it separates real tracking failures from task difficulty.
[2] The research questions are well designed and the method follows from them. The RQs are clearly articulated and answer the questions a reader is likely to have. RQ3 (does interference accumulate or persist) and RQ4 (can distillation help) are the most interesting and important. The Teacher-Student setup, with the Teacher matched to the active task, offers a way for the model to follow what is in effect without a Teacher at inference time. The table comparing single-turn and multi-turn conditions makes the results transparent.
Critical weaknesses
The realism of the inserted clarification and revision events is not fully established. Dialogues run about 4-8 turns, which is typical of real interactions. However, the proposals, rejections and clarifications are template-based, with LLM-drafted wording and only a human review. These events need to be examined to make sure they are natural and not out of order, since awkward insertions could confuse the model for reasons unrelated to intent tracking. The benchmark also covers only one revised requirement per task.
Questions for authors
[1] Have you encountered scenarios where the model provided a solution to both the changes that are in effect and the changes that were requested before but revised later? Will such kind of a response be considered incorrect in the current experimental setting?
[2] Can you explain why there is a relative increase in performance as event blocks k increases from 3 to 4 in Figure 5?
3 comments
Reply to Rebuttal
Thank you for your reply. I will keep my scores.
Thank you for taking the time to consider our responses. We appreciate your thoughtful review and feedback.
Response to Reviewer 5xwJ
Dear Reviewer 5xwJ,
Thank you for the careful review. We address your concerns as follows.
W1. Event realism and the scope of revisions.Q1. Responses containing both active and superseded solutionsYes, we observed this behavior, and a final response that executes both the active and superseded requests is scored incorrect. For example, after a user changed a factorial request from 5 to 7, the model included calls for both in its final answer. Since the accepted revision replaced the original request, the additional call for 5 violated the active requirements.
Q2. The increase from k=3 to k=4The increase most likely comes from the added block prompting the model to revisit which requirement remains active, which on some tasks lets it correct an earlier mistake. This increase is small and does not appear consistently across models and domains. Despite these local increases, average accuracy at k=4 remains below the corresponding k=0 baselines.
Thank you again for your positive assessment and constructive feedback. Your suggestions have helped us clarify the benchmark’s design, scoring criteria, and interpretation of the results. We hope these responses address your concerns and are happy to discuss any remaining questions.
Best,
The Authors