acceptodds
Under review as a conference paper at ICLR 2027

LLMs Get Lost in Evolving User Intent

Abstract

As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user-delegated tasks through iterative interaction. Yet genuine interaction is inherently dynamic: users rarely specify their intent upfront, instead disclosing, revising, and reshaping it as the conversation unfolds. Despite this, LLMs are still predominantly evaluated or trained in single-turn, fully-specified settings, leaving open a fundamental question: how well do LLMs track and act on user intent as it evolves over the course of a conversation? To study this, we introduce a framework that transforms static, single-turn tasks into dynamic multi-turn conversations in which the user's intent evolves across turns—incrementally revealed, revised, and at times redirected mid-conversation—while preserving each task's original evaluation protocol, enabling existing benchmarks to be reused as controlled testbeds without new annotation. Across multiple tasks, we surface a consistent phenomenon: strong static-setting performance does not transfer to the evolving-intent setting, with substantial drops across model families. Our findings point to a fundamental gap: today's LLMs do not yet faithfully track and act on the user's evolving intent, a capability invisible to static evaluation yet critical for future collaborative agents.

Primary area:
datasets and benchmarks
Keywords:
Multi-turn Evaluation, User Simulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rC9kOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper transforms verifiable single-turn benchmarks into multi-turn conversations with evolving user intent, including incremental information disclosure, argument revision, and task switching while preserving the original evaluation protocols. Experiments across four domains demonstrate substantial performance degradation under evolving intent, particularly with task switching, and show that existing memory and reasoning strategies do not fully mitigate these failures.

Critical strengths
  • Good writing with a clear motivation. Easy to understand.
  • I agree that converting a single turn task into the last turn is a controllable way to simulate multi-turn tasks.
  • Legitimate research questions with experimental supports.
  • The takeaway might be trivial because stress-testing language models may introduce additional challenges to solve given tasks. But appendix seems to show that changing intents actually decreases performance rather than simply increasing the number of turns, which is well covered by the authors.

I gave a positive score based on the reasons above, but I'm not championing it due to the following weaknesses.

Critical weaknesses
  • It shares some setup similarities with previous work (https://arxiv.org/abs/2505.06120), such as recycling original evaluators. Also, since controlled dialogue simulation with counterfactual values has existed before (e.g., https://arxiv.org/abs/2010.12850), the approach itself may not be novel enough.

  • This work evaluates whether LLMs can correctly solve a "final, verifiable" task after a synthetic sequence of information reveals, revisions, and task switches. Evaluating only a final step sacrifices some realistic assumptions.

Questions for authors

N/A

4 comments

Dear reviewer rC9k,

We sincerely appreciate your efforts and insightful comments to improve the manuscript, as well as your overall positive review. We respond to each of your comments one by one in what follows. All revisions in the manuscript are highlighted in blue.


[W1] Similarities with prior works.

While we share similarities with prior works [1,2], we have tackled the key challenge that previous simulation works struggle with: long-horizon user-LLM interaction with workflow verification.

Unlike [1], our framework can model long-horizon user interactions through retrospective expansion. Specifically, we generate plausible earlier requests and simulate how the user arrives at the source problem through a sequence of argument revisions and function switches. In contrast, [1] splits the source problem into parts and reveals them over successive turns. The number of disclosure turns is therefore limited by how many details can be revealed separately, so a short search query with no separable details cannot yield a multi-turn interaction through sharding alone.

Furthermore, our ultimate goal is to verify whether an LLM can complete the user’s workflow as their intent evolves. While [2] also uses a latent intent to control the simulated user, its evaluation focuses on whether a dialogue state tracker recovers that intent at each turn. To measure workflow success beyond turn-level intent recovery, our framework transforms verifiable tasks from realistic workflows into multi-turn conversations. This distinction matters in practice: as shown in Section 5.3, accurate turn-level intent tracking does not necessarily lead to final task success, since agents often recover the new intent right after a function switch yet fail to integrate it with the earlier context.

[1] LLMs Get Lost In Multi-Turn Conversation, ICLR 2026
[2] CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers, ICLR 2021


[W2] Only verify in the final turn.

We agree with the reviewer that intermediate verification would be valuable for other purposes, for example as a process reward for training. For our goal of measuring the effect of evolving intent, final-turn verification provides a clear signal. We therefore make the main comparison at the final turn and complement it with analyses of intermediate behavior.

The comparison requires evaluating the same intent in both settings, and the accumulated intent matches the single-turn problem at the final turn. Intermediate intents are different problems with different difficulties. For example, if a hard condition is revealed late, the intermediate problems are much easier than the final one, so final-turn evaluation gives a fairer comparison against the single-turn reference. To measure how performance changes over the course of a conversation while keeping the problem fixed, we vary the number of transitions, as reported in Figure 4.

For intermediate behavior, our turn-wise intent tracking analysis (Table 3) prompts the model to predict the current user intent at every turn. Tracking accuracy is lowest on function switches and degrades as transitions accumulate. In addition, the newly added reasoning traces of Kimi K2.6 (Appendix E.5) show how the accumulated conversation affects the agent's reasoning: under evolving intent, the model keeps revisiting earlier requests and shows more than five times as many hesitation markers as in the single-turn setting.


Dear Reviewer rC9k,

Thank you again for your time and efforts in reviewing our paper.

With two days remaining in the discussion period, we would welcome any further comments or questions you may have. We would appreciate the opportunity to address any remaining concerns before the discussion period ends.

Thank you again!

Best regards,
Authors

Thanks for the response. I'd like to keep my original positive score.

Dear Reviewer rC9k,

Thank you again for your time and continued positive assessment. We are glad our rebuttal addressed your concerns.

Please feel free to raise any further questions during the remaining rebuttal period.

Best regards,
Authors

Reviewer_y8NiOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a simulation framework that transforms various single-turn tasks into multi-turn conversations where the user's intent evolves dynamically across turns. The experiments show that LLMs perform worse when the user's intent becomes dynamic.

Critical strengths
  1. Extensive experiments demonstrate that LLMs perform worse when the user's intent becomes dynamic.
  2. Ablation studies and error analysis further support the claims of the paper.
Critical weaknesses
  1. Lack of in-depth case studies to investigate how LLM agents change their behavior once the intent becomes dynamic.
  2. Some questions and concerns, as in "Questions For Authors".
Questions for authors
  1. Is there quality control on the simulated or transformed dynamic intents in multi-turn conversations?
  2. How do the key configurations during the benchmark transformation affect model performance?
  3. Results: How many runs were there for each reported score? What is the statistical significance and confidence?
3 comments

Dear reviewer y8Ni,

We sincerely appreciate your efforts and insightful comments to improve the manuscript, as well as your overall positive review. We respond to each of your comments one by one in what follows. All revisions in the manuscript are highlighted in blue.


[W1] How do agents change their behavior once the intent becomes dynamic?

We first remark that the paper already analyzes how agent behavior changes under dynamic intent. (i) Agents become indecisive: on SWE-Bench, agents exhaust the budget of 100 tool calls per turn, where fewer than 4 calls are execution-related (e.g., pytest), and the rest are exploration (e.g., grep) (Section 5.1). (ii) Agents lose track of the intent: turn-wise intent tracking remains nearly perfect under argument reveals and revisions, but drops to 82% under function switches (Table 3). (iii) Agents fail to manage their context: even when the oracle intent is restated at every turn, accuracy stays below the single-turn level (Figure 6).

Nevertheless, we agree that trajectory-level case studies would make these findings even more concrete. To this end, we have added the reasoning traces of Kimi K2.6 on the same BrowseComp+ problem under both single-turn and evolving intent in Appendix E.5 (we use an open-source model, as its reasoning is fully accessible). As shown, under evolving intent, the model becomes confused about what the user currently wants: its final-turn reasoning contains more than five times as many hesitation markers (e.g., “wait”, “maybe”, and “but the user”) and in some case about 17 times more reasoning tokens than in the single-turn setting. This is consistent with the overall trend that models spend more tokens under evolving intent (Figure 9), which suggests increased uncertainty.


[Q1] How to control the quality of the simulated conversations?

Yes, we apply quality control at every stage of the transformation (Appendix C and D). (i) Intent construction: every LLM-generated component is verified by rejection sampling. The extracted intent must cover the source question and yield the original answer, each counterfactual argument must be a localized value substitution, and each predecessor function must not alter the answer of the final task. (ii) Scheduling: transitions are scheduled under consistency rules (Section 4.3), which guarantee that the conversation ends at the source intent with all arguments revealed. (iii) Rendering: each user turn is rendered by a rule-based renderer, which preserves every value exactly and prepends a discourse prefix that explicitly signals the transition type (e.g., “Wait, I made a mistake.” for a revision, and “Instead, I’d like to know:” for a function switch). Hence, the intent change in each turn is unambiguous to the agent. When the LLM-based naturalizer is used, we check that all critical values are preserved, and fall back to the rule-based turn otherwise.


[Q2] How do the key configurations during the benchmark transformation affect model performance?

We carefully remark that the paper already studies the key configurations of our transformation: (i) the number of transitions of each type (Figure 4), (ii) the composition of transition types (Table 2), (iii) the order of transitions, i.e., whether further transitions follow a function switch (Figure 5), (iv) the number of conversation turns (Appendix E.4, Table 6), and (v) the difficulty of the source data (Figure 7). Overall, accuracy degrades monotonically as more transitions are added, and the degradation is driven by the intent transitions rather than the number of turns.


[Q3] How many runs are used to report the score, and what is the statistical significance?

Each score is reported from a single run. However, we remark that our findings are reliable for two reasons. First, the degradation is consistent across models and domains: accuracy drops in 35 of 36 model-dataset pairs in Table 1. Second, our simulation is deterministic (i.e., the user turns are fixed for each instance) and we use greedy decoding where supported, so the only source of variance is the model itself.

Nevertheless, to quantify this variance, we conducted 4 additional runs of GPT 5.1 on GSM8K with sampling enabled (API default temperature), in addition to the original run. The results are highly stable: the standard deviation across the 5 runs is 0.65, which is far smaller than the 16% gap between the single-turn and evolving settings.


Reviewer_y8NiOpenReview ↗

Rebuttal Acknowledgement by Reviewer y8Ni

Thank you for the reply. Please include the key points in the revision and explicitly refer to the corresponding appendices in the main body text. Overall, I maintain a positive rating of the submission.

Thank you again for your time and continued positive assessment. We are glad our rebuttal addressed your concerns. We have incorporated all the points you raised into the revised paper, including explicit references to the corresponding appendices in the main text, and will retain these changes in the final version.

Please feel free to raise any further questions during the remaining rebuttal period.

Best,
Authors

Reviewer_gNikOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether LLM agents can keep tracking and acting on the user's current intent as the user adds constraints, revises them, and switches tasks. Starting from verifiable single-turn tasks, the framework fixes the source task's intent as the last-turn anchor. It uses retrospective expansion to generate counterfactual arguments and predecessor functions, then schedules and renders them into multi-turn conversations. The final evaluation uses the original task's verifier. The experiments show lower accuracy for Evolve than for Single in most tested configurations. For example, GPT 5.5's accuracy on GSM8K drops from 99.0% to 80.5% after six intent transitions. The paper also examines reasons for these drops and possible ways to reduce them through analyses of transition types and combinations, memory mechanisms, turn-wise intent tracking, and preliminary RL experiments.

Critical strengths

S1. The framework makes intent changes a variable that can be controlled independently. It clearly distinguishes argument reveal, argument revision, and function switch, and uses a scheduler to control the number and mix of transitions. The experiments compare the effects of these types on their own and in combination, giving a practical way to test which intent changes are harder for models to handle.

S2. The main results cover both answer generation and tool use across a range of settings. The main experiments cover GSM8K, BIRD-SQL, BrowseComp+, and SWE-Bench Verified, as well as nine models, spanning math, text-to-SQL, agentic search, and software engineering. This setup allows the paper to examine difficulties in both answer generation and tool interaction. It provides evidence across models and tasks for performance drops under synthetic evolving-intent interactions.

Critical weaknesses

W1. Verification of the complete generated conversations is insufficiently documented. Appendix D uses GPT 5.1 to verify the generated components, while a validator checks whether utterances produced by the LLM-based naturalizer preserve critical tokens. The paper does not report a manual audit of the generated conversations or a check that, after all turns, the final task is unambiguous and matches the source task. For SWE, it states that the final rendered turn is checked for alignment with the original issue, without describing the procedure. This leaves uncertainty about whether dialogue ambiguity contributes to the observed degradation.

W2. The paper doesn't provide enough information and resources to reproduce and reuse the evaluation framework. Appendix F explicitly omits prompts and few-shot examples for the other domains. Appendix C also does not give the full prefix inventory or the LLM-based naturalizer prompt. The paper doesn't list the exact evaluation samples or provide a way to access the code or generated conversations. These details directly affect how the conversations are constructed and how difficult they are. The current appendices aren't enough to reconstruct the evaluation set and generation process used in the paper.

W3. The sharp performance drops on SWE are affected by tool budgets and timeouts, and the budgets differ across models. Most models have a per-turn tool-call budget of 100, while Kimi K2.6 and DeepSeek V3.2 have 200. The authors explicitly note that GPT 5.1 and some other models exhaust the budget and time out, but they don't report a comparison under the same budget or report timeouts separately from incorrect final patches. The SWE results reflect both how well models adapt to the task and the resources they are given. It is hard to attribute these drops directly to intent tracking failures, and unequal budgets also limit comparisons across models.

Questions for authors

Q1. Does Single use the original single-turn source problem or a fully specified rendering of the extracted intent? For BIRD-SQL, the intent is constructed directly from gold SQL, but the exact input used for Single is unclear. Please clarify this input and consider a single-turn control that presents the final function and all arguments using the renderer employed for Evolve. This would help distinguish changes in input wording from the effects of evolving intent.

Q2. How do the results in Table 2 support the generalizations in §5.2? The paper states that function switch “consistently emerges as the most challenging transition type” and that degradation becomes more pronounced when other transitions are added. However, GPT 5.5 achieves 46% accuracy under Argument reveal versus 62% under Function switch on BrowseComp+. On BIRD-SQL, both evaluated models achieve higher accuracy under Reveal + Revise + Switch than under Function switch. Please explain these differences and clarify the domains and models to which these claims apply.

6 comments

Dear reviewer gNik,

We sincerely appreciate your efforts and insightful comments to improve the manuscript. We respond to each of your comments one by one in what follows. All revisions in the manuscript are highlighted in blue.


[W1] Insufficient details in Appendix: Details of verification.

We clarify how we construct conversations and verify intents to ensure that the final intent matches the source task. First, situated simulation guarantees that the full conversation converges to the source task. A scheduler determines the sequence of user turns: the last function switch returns to the source function, and the last revision of each argument restores its source value (for easy explanation; the accumulated intent overwrites previous functions arguments, thus ending with the source intent by design). The final intent therefore matches the source task by design across all domains, including SWE.

This guarantee relies on preserving the source information and preventing newly introduced arguments from interfering with the source task. For instance, for math, we verify that (i) the extracted problem preserves the source information and an LLM’s predicted answer matches either the gold answer or its prediction for the original problem; (ii) each argument revision replaces only one value, checked programmatically; and (iii) added predecessor arguments do not leak information across turns, and an LLM’s predictions for the source task with and without these arguments agree, or the prediction with them matches the gold answer. Samples that fail are regenerated or excluded.

The per-domain procedures are detailed in the revised Appendix D, and the full pipeline is available in the released code.


[W2] Insufficient details in Appendix: Details of code/prompt

Thank you for raising this point. To ensure full reproducibility, we have released our complete code and attached it to the submission. It contains the prompts and few-shot examples for all domains, the full prefix inventory, and the LLM-based naturalizer prompt. We omitted these from the appendix only because of their length. The evaluation samples were selected at random, and the exact sample indices are included in the code, so the evaluation set used in the paper can be reconstructed exactly.

In addition, our release is a simulation framework rather than a fixed evaluation set. The same code can generate additional evaluation conversations, or convert training splits into multi-turn data for training.


[W3] Sharp performance drops on SWE are from tool call budgets.

We would like to clarify that our central comparison is between the single-turn and evolving-intent settings for the same model, under an identical per-turn budget (in total evolving uses 7 times more tool calls). A tool-call budget is standard practice for evaluating SWE agents, and our default of 100 calls follows prior work [1,2]. We raised it to 200 only for Kimi K2.6 and DeepSeek V3.2, because 100 calls constrained them even in the single-turn setting (e.g., Kimi K2.6: 54% with 100 vs. 86% with 200).

Nevertheless, we agree that the role of the budget should be made explicit, so we reran GPT 5.1 with 200 tool calls per turn. It still exhausts the budget on all instances, and accuracy remains 0.0, so a larger budget does not recover the drop. Importantly, these failures are not incorrect final patches: the model never submits a patch, and the timeouts we mention are exactly this budget exhaustion. As shown in Section 5.1, the model keeps gathering information instead of acting (over 96 of 100 calls per turn are search), although the same source problems are solved in about 20 calls in the single-turn setting. We believe this is itself a key finding: under evolving intent, the agent appears uncertain about how to proceed and keeps gathering information instead of committing to the final action.

[1] R2E-Gym: Procedural Environment Generation and Hybrid Verifiers for Scaling Open-Weights SWE Agents, COLM 2025
[2] SWE-bench Goes Live! NeurIPS 2025


[Q1] Does Single use original single-turn prompt (or reconstructed prompt from extracted intent).

We clarify that Single already uses the control the reviewer suggests: the same extracted intent and the same renderer as Evolve, with the final function and all arguments presented in one turn. For BIRD-SQL, the function is the original question with its condition values removed, the arguments are the conditions parsed from the gold SQL, and the schema and evidence are prepended to the first turn. Hence, the gap between Single and Evolve is not due to input wording.

Nevertheless, we also compared this rendering against the original BIRD-SQL question and obtained identical accuracy for GPT 5.1 (72.0% for both). We have clarified the Single input in the revision.


[Q2] How do the results in Table 2 support the generalizations in §5.2?

We agree with the reviewer and have revised “consistently” to “generally,” since the ordering varies across domains and models. Nevertheless, the general claim is well supported by Table 2: across the 8 domain-model pairs, function switch lowers accuracy in all 8, causes the largest drop among the three transition types in 6, and has the largest average drop (16.7 points, vs. 7.4 for argument reveal and 2.3 for argument revise).

The hardest transition can naturally differ across domains and models, but the key finding is that models struggle more under evolving intent than in the single-turn setting. For example, agentic search depends on the retriever as well as the LLM, so an under-specified query may retrieve less relevant documents and shift the pattern. In BIRD-SQL, a function switch changes the query structure itself (i.e., the main clauses such as SELECT and GROUP BY), which may be the hardest part of the task and thus account for most of the drop, while the additional interactions in the composed settings may introduce only small variation around it.


Dear Reviewer gNik,

Thank you again for your time and efforts in reviewing our paper.

With two days remaining in the discussion period, we would welcome any further comments or questions you may have. We would appreciate the opportunity to address any remaining concerns before the discussion period ends.

Thank you again!

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.