- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the challenges of maintaining accurate state information in LLMs, particularly when outdated but plausible information remains in the history and influences current decisions. The authors introduce a benchmark comprising 1,200 instances designed to isolate the effects of stale information through controlled experimental conditions. The proposed method, STATEFLOW, maintains valid state information externally and provides the model with the resolved current state. Experimental results demonstrate the effectiveness of the proposed approach, showing substantial improvements over the baselines.
Critical strengths
- This research addresses an important challenge in understanding and reasoning over long historical contexts in LLMs. The motivation is clear and well justified.
- The experimental setup is well designed, as it systematically distinguishes among stale-semantic interference, explicit state representation, and history isolation. The experimental findings provide a solid foundation for the proposed STATEFLOW approach.
- The introduced benchmark dataset is also a valuable contribution and could serve as a useful resource for the research community.
Critical weaknesses
- The generalizability of the proposed method has not been fully explored. While STATEFLOW demonstrates clear benefits in the experimental setting involving Jira tasks with diverse histories, it remains unclear whether the approach can be effectively applied to other domains requiring long-context understanding, such as long-context question answering or personal AI agents for everyday tasks.
- The models evaluated in Table 1 appear to be somewhat outdated. It would be valuable to evaluate more recent state-of-the-art models on the proposed benchmark to assess whether the identified challenges persist in newer models.
Questions for authors
.