acceptodds
Under review as a conference paper at ICLR 2027

DynaEval: Large Language Models Struggle to Track Evolving Task States

Abstract

Large language model evaluation has largely focused on static, single-turn tasks. Real-world interactions, however, often involve multi-turn conversations in which task requirements evolve. We evaluate model performance in this setting and find that accuracy declines across dynamic multi-turn conversations. We attribute this decline to limitations in models’ ability to reconstruct the current task state from information dispersed across the conversation history. To evaluate this ability, we introduce DynaEval, a benchmark that simulates dynamic, long-horizon task environments. DynaEval comprises 543 eleven-turn sequences across mathematics, code, and databases, and provides two aligned versions of each turn: one presents an incremental update to the task conditions, while the other provides a complete description of the conditions valid after that update. Both versions share the same target question and reference answer. DynaEval comprises 543 eleven-turn sequences across mathematics, code, and databases. Under the full-history protocol with incremental updates, all ten models exhibit lower final-turn than first-turn accuracy, with the mean falling from 67.9% to 32.9%. Further diagnostic experiments attribute this cross-turn accuracy decline primarily to models’ difficulty in recovering the currently valid task conditions from information dispersed across prior interactions, revealing a bottleneck in multi-turn conversations and long-horizon tasks.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Dynamic Multi-Turn Evaluation, State Tracking and Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x71TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission introduces DynaEval, 543 eleven-turn conversations (math, code, SQL) in which the user keeps revising the task. Every turn comes in two versions, the change alone and the full current task. The authors use it to argue that models get worse in later turns because they fail to work out which conditions currently hold, which they call state reconstruction.

Critical strengths

Having both versions of every turn makes it easy to see how much is lost when the model has to track the changes itself.

Critical weaknesses

The key evidence is that accuracy goes back up (43.2% to 65.2% on seven models) once the current task is written out for the model, but writing it out is exactly the step under test, so this is expected and says nothing about why the model fails at that step. What I really miss is a simple baseline: ask the model to restate the current task first, then answer. If that recovers most of the gap, the models can do it and just don't, and I wouldn't call it a capability problem.

Questions for authors

Did you try having the model restate the current task before answering? How much of the gap does it close?

Reviewer_zbeUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work analyzes the degradation of model accuracy in dynamic multi-turn conversations. The paper decomposes each turn into "state reconstruction" and "task solving", and constructs 543 eleven-turn tasks across 3 domains. Under the full-history protocol, the mean accuracy across 10 models drops from 67.9% at the first turn to 32.9% at the final turn, and every model scores lower at the final turn. Through injecting different types of state information, the paper finds that providing the resolved current state restores accuracy to near oracle levels.

Critical strengths
  1. The paper decomposes multi-turn degradation into state reconstruction vs. task solving, and isolates the two via paired incremental/self-contained turns sharing the same answer. The conclusion is credible because alternative explanations are excluded one by one.
  2. Model rankings depend on the evaluation horizon (Claude Sonnet 4.6 has the highest mean FH accuracy, but GPT-5.4 the highest final-turn accuracy), directly challenging single-turn leaderboards. The repeated injection result is also a proof of concept for agent frameworks.
Critical weaknesses
  1. The RR control is a fixed passive reminder that never asks the model to externalize reconstruction (e.g., list each parameter's current value and source turn before answering). So "cannot reconstruct" and "does not reconstruct spontaneously" remain undistinguished.
  2. All three domains admit executable answer checking. Trajectories rely on hand-designed operator updates with templated wording, and the 543 sequences required human review, making extension to new domains costly.
Questions for authors
  1. The RR control is a fixed passive reminder. Could you test a stronger version that asks the model to explicitly reason before answering (such as first list each task parameter's current value and the turn that established it, then answer the current request). If such explicit reconstruction recovers a large share of the ES gain, the bottleneck may be spontaneity rather than capability, which would substantially change the practical implications of the paper.
  2. Could you run a wrong-ES control which means an ES block with identical format and instructions but an incorrectly resolved state? This would quantify how much of the ES gain comes from the resolved state information itself versus from the block shielding the model from history interference.
Reviewer_1uCSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark named DYNAEVAL that simulates dynamic, long-horizon environments. The benchmark includes intensive sequences across three tasks and evaluates model performance in multi-turn conversations. Through experiments and analyses, the paper shows that model performance declines as the number of conversational turns increases. Furthermore, the authors evaluate several methods intended to mitigate this issue and provide relevant analyses.

Critical strengths

The paper is well motivated, as it addresses the important question of performance degradation in long-horizon conversations. The authors conduct extensive experiments to support their findings and evaluate several methods intended to mitigate performance degradation.

Critical weaknesses

The paper contains some rough, apparently unverified text. For example, the abstract describes the composition of the benchmark twice using the same sentence. Some terms, such as OST, RR, HR, and ES, are not explained clearly, and the relevant sections do not include links to the appendix. The paper presents results only on incremental-state data but does not report results on self-contained data. Following the previous point, in the incremental-state setting, the question changes at each turn, so its difficulty may also vary across turns. The authors should additionally report results on the corresponding self-contained data and evaluate how model performance changes from the self-contained setting to the incremental-state setting.

Questions for authors

Please find the questions shown in Weakness.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your reviews and contributions. As we are now halfway through the discussion period, I would like to encourage you to carefully read the authors' responses and engage in further discussion where appropriate. Please also indicate whether the rebuttal has addressed your concerns or changed your assessment.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.