- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies if coexisting descriptions of a "task state" impair the downstream results, even if the descriptions are identical. The authors conducted controlled experiments on this problem and the results reveal that current models can roughly identify updated facts yet still fail to use them, and even repeating identical information would negatively affect their performance. Rewriting the context into a single source-of-truth description often helps but can also affect performance on longer horizon when the rewrite itself induces errors.
Critical strengths
- the central claim of the paper is well supported by the results, and the controlled experiments were relatively well executed
- the authors also report failed compilation/etc cases and are honest about the boundary of these experiments and attempted solutions, which should be appreciated
Critical weaknesses
The main complaint is the controlled experiment might imply a narrower scope than the headline suggest and primarily limited to the models that have been evaluated. If such a narrowly scoped observation study merits enough contribution per ICLR standard. To push it further is if such kind of observation study can offer deeper insights to the community on how we could amend this type of failure in real-world LLM uses.
Update: Thank you to the authors for the detailed response and the effort of updating the manuscript and adding new information about my question. The new budget comparison has addressed one of my concerns and I appreciate the effort to revise and sharpen the intro and abstract. Therefore I've increased my overall recommendation score.
Questions for authors
From the reasoning budget lens, does rewriting the context before answering still outperform, e.g., simply letting the model reason longer?
2 comments
Please engage with the authors during the discussion period.
Dear reviewer, The authors have provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale.
Thank you for your positive feedback and for reconsidering your assessment. We appreciate your time and thoughtful review.