- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the gap between retrieving relevant evidence and using it appropriately in memory-based language-model agents. It evaluates two dimensions separately: task satisfaction and whether the entire response is supported by model-visible evidence. Joint passage requires both. The evaluation includes retrieval diagnostics for six memory systems, history comparisons across 54 stories and three model configurations, evidence removal in 13 stories, 162 explicit-state scenarios, and 106 archived τ-bench retail interactions. The results show that complete retrieval does not guarantee appropriate actions, key state records improve task satisfaction, and native task success can diverge from final-response grounding.
Critical strengths
- A useful evaluation distinction.
Separating task satisfaction from whole-response grounding distinguishes answers that meet the target without adequate support from cautious, justified answers that leave the task incomplete. This helps identify failures beyond retrieval and gives appropriate attention to action recommendations and status claims.
Critical weaknesses
- Some task references impose requirements that are not uniquely determined by the visible input.
Only 2 of the 54 unknown-receipt inputs explicitly require checking prior delivery; the other 52 require interpretation. The authors acknowledge that asking for fresh resend permission may be reasonable, yet the strict reference marks it as task-incomplete. These cases account for many failures by Qwen 35B and Gemma. Without consistently applying an alternative rubric to all relevant responses, it is difficult to determine how much of the reported gap reflects model behavior versus the chosen reference interpretation.
- The central findings depend on semantic annotations whose independence and robustness remain insufficiently established.
The paper documents initial reviews, AI-assisted coordination, and later adjudication, while acknowledging gaps in reviewer exposure histories and identity-to-version records. Initial agreement is encouraging, but does not rule out shared interpretation errors. Several sensitivity analyses apply only to an earlier score release. Since measurement is the paper’s main contribution, uncertainty about the current annotations directly limits its persuasiveness.
Questions for authors
-
Could you apply a clearly defined alternative rubric accepting a reasonable next step to all 162 unknown-receipt responses, and report the task and joint passage for each configuration? This would materially affect my interpretation of the explicit-state results.
-
For the scores used in the main figures, could you clearly identify which judgments came from independent initial reviews, AI-assisted coordination, and later adjudication, along with the information available at each stage? Could anonymized item-level judgments and change rationales be made available?
-
Could you provide sensitivity analyses for disputed judgments in the current score release? In particular, do the history gains, evidence-removal effects, and explicit-state differences remain robust under plausible alternative judgments?