- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TaskFid evaluates whether agent memory preserves and updates task state across sessions and threads, using questions tied to specific points in an evolving task history.
Critical strengths
Frozen-Anchor evaluation tests the same memory state without feeding answers or judge feedback back into memory.
Critical weaknesses
-
It acknowledges missing independent checks of reference answers and judge accuracy. Appendix D does not measure agreement among independent readers. Wrong references could count correct memory responses as failures. TaskFid needs this check for its own questions. The current scores does not separate memory errors from evaluation errors.
-
It excludes missing or invalid judge results, which could leave methods scored on different query sets. This concerns judge failures, not automatic removal of badly formatted answers. If difficult cases are excluded unevenly, rankings may change. Equal condition weights do not fix this. The exclusion rule need missing-rate reports, comparisons on shared judged queries, and bounds for unresolved cases.
-
It scores ten source messages, while the answerer receives all retrieved text. The payment example places useful RAG evidence beyond this cutoff. The retrieval–answer contrast need metrics covering actual answerer input and factual content preserved in summaries.
-
It credits later-correct unchanged fields even when they were wrong earlier. This mixes preservation with new recovery. TaskFid similarly needs to distinguish retained ability from newly recovered facts. The retention label make this unclear. Report correct-to-correct, correct-to-wrong, and wrong-to-correct field transitions separately to show what was actually retained.
-
Oracle loses 26.2 in Section 4.4, while Appendix C.8 shows later answers requiring larger maps. Declining scores can therefore reflect harder work rather than forgetting. These curves needs controls for answer size and evidence placement, plus separate tracking of previously correct, unchanged facts.
Questions for authors
Please refer to the weaknesses.