- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework called MarkovLens for diagnosing failures in task state forLLM agents. It distinguishes interaction history from the current “task state”, and evaluate whether models can use the state to make the correct decisions. Experiments on Qwen and DeepSeek models show that providing correct state can substantially improve accuracy and make decisions more consistent.
Critical strengths
- It is insightful to differetite the distinction between correcting an error, following decision-relevant changes, and ignoring answer-preserving changes. This is beyond a simple task accuracy and is a more informative of evaluation.
- The experiments are carefully designed, e.g., using equivalent histories and matched counterfactual interventions to isolate the effect of state information.
- The results are supported by experiments with different inference settings.
Critical weaknesses
- The evaluation is mostly based on synthetic tasks with predefined, deterministic state transitions and answers. It is unclear how much the findings generalize to real-world agentic interactions, where the current state may be ambiguous or not observable. Although the paper includes semi-natural language experiments, these are still generated from the same underlying tasks.
- Related to the previos point, the framework assumes that the correct task state is readily available and can be provided to the model. However, the agent needs to extract and maintain this state from the interaction history. Table 10 shows that state extraction performance drops significantly under unseen values. Therefore, it is unclear how much the proposed framework can help diagnose real-world agents when the correct state is not readily available.
- The main matched-intervention experiments focus on user constraint update, while the other two task families do not have extensive evaluation.
Questions for authors
- Can MarkovLens be applied to real-world agent benchmarks where the current state is not predefined?
- Would the same repair patterns hold when the state is extracted by an LLM rather than provided as ground truth?