- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an event-level framework ORPMTF to diagnose task progress state mismatch for long-horizon tool-use agents, decoupling two dimensions of state-update validity. The annotated trajectory analysis reveals two fundamental types of mismatch and provides fine-grained insights for state auditing and agent error recovery.
Critical strengths
- Existing agent evaluations mostly focus on the final task success/failure and tool‑call errors. This paper shifts the focus to intermediate task‑state management. It points out a key phenomenon: a successful individual tool operation does not guarantee that the prerequisites for downstream tasks are satisfied. Conversely, an operation error does not necessarily result in mismanaged task states. This perspective is valuable for research on the reliability of long‑horizon agents.
- The framework is clearly defined with rigorous conceptual distinctions. It decomposes state alignment into two dimensions: whether updates go beyond permitted bounds, and whether mandatory required changes are omitted.
- The evaluation corpus is large‑scale and covers multiple environments. The appendix provides comprehensive details to improve readibility.
Critical weaknesses
- The paper explicitly states that these are weak labels, and the planned independent human validation described in Appendix G is only a proposal and has not been executed. Quantitative inter‑annotator agreement metrics between machine labels and human labels are missing. It remains unknown how large the discrepancy is between machine‑judged P/M/T and human expert judgments.
- The 600 manually reviewed development‑set trajectories cannot serve as validation data, since they were used to design rules during framework development, which introduces the risk of overfitting.
- Experiments are mainly conducted on the GPT series, DeepSeek, MiMo and MiniMax. Mainstream open‑source and closed‑source models including Claude, Kimi and GLM are absent. Given the differences across models in reasoning style, state memory and hallucination tendency, it is unclear whether the two types of mismatch observed in this work generalize to other model families.
- Some in‑depth analyses and discussions are missing: why does the incidence of state mismatch vary drastically across different task environments? What task properties are more likely to trigger progress overreach or mandatory‑update omission?
- More intuitive illustrative examples for P/M/T/F would benefit the main text, as these concepts are relatively hard to follow.
Questions for authors
Please thoroughly address all the weaknesses raised above in your rebuttal. Where necessary, update the main paper and appendices to reflect corresponding revisions.
Dear Reviewers,
I hope this message finds you well.
I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.
Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.
If you have any questions or need further assistance, please do not hesitate to reach out.
Best regards,
AC