- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper transforms verifiable single-turn benchmarks into multi-turn conversations with evolving user intent, including incremental information disclosure, argument revision, and task switching while preserving the original evaluation protocols. Experiments across four domains demonstrate substantial performance degradation under evolving intent, particularly with task switching, and show that existing memory and reasoning strategies do not fully mitigate these failures.
Critical strengths
- Good writing with a clear motivation. Easy to understand.
- I agree that converting a single turn task into the last turn is a controllable way to simulate multi-turn tasks.
- Legitimate research questions with experimental supports.
- The takeaway might be trivial because stress-testing language models may introduce additional challenges to solve given tasks. But appendix seems to show that changing intents actually decreases performance rather than simply increasing the number of turns, which is well covered by the authors.
I gave a positive score based on the reasons above, but I'm not championing it due to the following weaknesses.
Critical weaknesses
-
It shares some setup similarities with previous work (https://arxiv.org/abs/2505.06120), such as recycling original evaluators. Also, since controlled dialogue simulation with counterfactual values has existed before (e.g., https://arxiv.org/abs/2010.12850), the approach itself may not be novel enough.
-
This work evaluates whether LLMs can correctly solve a "final, verifiable" task after a synthetic sequence of information reveals, revisions, and task switches. Evaluating only a final step sacrifices some realistic assumptions.
Questions for authors
N/A
4 comments
Dear Reviewer rC9k,
Thank you again for your time and efforts in reviewing our paper.
With two days remaining in the discussion period, we would welcome any further comments or questions you may have. We would appreciate the opportunity to address any remaining concerns before the discussion period ends.
Thank you again!
Best regards,
Authors
Thanks for the response. I'd like to keep my original positive score.
Dear Reviewer rC9k,
Thank you again for your time and continued positive assessment. We are glad our rebuttal addressed your concerns.
Please feel free to raise any further questions during the remaining rebuttal period.
Best regards,
Authors
Dear reviewer rC9k,
We sincerely appreciate your efforts and insightful comments to improve the manuscript, as well as your overall positive review. We respond to each of your comments one by one in what follows. All revisions in the manuscript are highlighted in blue.
[W1] Similarities with prior works.
While we share similarities with prior works [1,2], we have tackled the key challenge that previous simulation works struggle with: long-horizon user-LLM interaction with workflow verification.
Unlike [1], our framework can model long-horizon user interactions through retrospective expansion. Specifically, we generate plausible earlier requests and simulate how the user arrives at the source problem through a sequence of argument revisions and function switches. In contrast, [1] splits the source problem into parts and reveals them over successive turns. The number of disclosure turns is therefore limited by how many details can be revealed separately, so a short search query with no separable details cannot yield a multi-turn interaction through sharding alone.
Furthermore, our ultimate goal is to verify whether an LLM can complete the user’s workflow as their intent evolves. While [2] also uses a latent intent to control the simulated user, its evaluation focuses on whether a dialogue state tracker recovers that intent at each turn. To measure workflow success beyond turn-level intent recovery, our framework transforms verifiable tasks from realistic workflows into multi-turn conversations. This distinction matters in practice: as shown in Section 5.3, accurate turn-level intent tracking does not necessarily lead to final task success, since agents often recover the new intent right after a function switch yet fail to integrate it with the earlier context.
[1] LLMs Get Lost In Multi-Turn Conversation, ICLR 2026
[2] CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers, ICLR 2021
[W2] Only verify in the final turn.
We agree with the reviewer that intermediate verification would be valuable for other purposes, for example as a process reward for training. For our goal of measuring the effect of evolving intent, final-turn verification provides a clear signal. We therefore make the main comparison at the final turn and complement it with analyses of intermediate behavior.
The comparison requires evaluating the same intent in both settings, and the accumulated intent matches the single-turn problem at the final turn. Intermediate intents are different problems with different difficulties. For example, if a hard condition is revealed late, the intermediate problems are much easier than the final one, so final-turn evaluation gives a fairer comparison against the single-turn reference. To measure how performance changes over the course of a conversation while keeping the problem fixed, we vary the number of transitions, as reported in Figure 4.
For intermediate behavior, our turn-wise intent tracking analysis (Table 3) prompts the model to predict the current user intent at every turn. Tracking accuracy is lowest on function switches and degrades as transitions accumulate. In addition, the newly added reasoning traces of Kimi K2.6 (Appendix E.5) show how the accumulated conversation affects the agent's reasoning: under evolving intent, the model keeps revisiting earlier requests and shows more than five times as many hesitation markers as in the single-turn setting.