- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This submission introduces DynaEval, 543 eleven-turn conversations (math, code, SQL) in which the user keeps revising the task. Every turn comes in two versions, the change alone and the full current task. The authors use it to argue that models get worse in later turns because they fail to work out which conditions currently hold, which they call state reconstruction.
Critical strengths
Having both versions of every turn makes it easy to see how much is lost when the model has to track the changes itself.
Critical weaknesses
The key evidence is that accuracy goes back up (43.2% to 65.2% on seven models) once the current task is written out for the model, but writing it out is exactly the step under test, so this is expected and says nothing about why the model fails at that step. What I really miss is a simple baseline: ask the model to restate the current task first, then answer. If that recovers most of the gap, the models can do it and just don't, and I wouldn't call it a capability problem.
Questions for authors
Did you try having the model restate the current task before answering? How much of the gap does it close?
Dear Reviewers,
Thank you for your reviews and contributions. As we are now halfway through the discussion period, I would like to encourage you to carefully read the authors' responses and engage in further discussion where appropriate. Please also indicate whether the rebuttal has addressed your concerns or changed your assessment.
Thank you for your time and effort!