- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether selecting procedural priors by query relevance affects the performance of an LLM agent’s memory retrieval planner. Using SimpleMem, it compares relevance-selected priors, randomly selected priors from the same bank, and no priors, while measuring variability across repeated executions. The experiments find no consistent performance difference between selection strategies; the most extensively repeated configuration excludes an effect as large as the earlier reported result, while the other configurations remain inconclusive. The paper emphasizes controlled comparisons and repeated evaluation before interpreting small performance differences as evidence of positive or negative transfer.
Critical strengths
- Holding the bank, adapter, budget, memory store, and injection point fixed isolates the effect of the selection rule.
- The paper distinguishes excluded effects from inconclusive results, identifies single-run checks, and documents failed executions and sensitivity analyses.
Critical weaknesses
- Generality and the distinct contribution need stronger support. The null result is established primarily for one memory system using deliberately out-of-domain priors. Please include a task-relevant prior bank as a positive control and repeat the relevance/random/no-prior comparison under the same budget. This would distinguish an ineffective selection rule from an injection mechanism that ignores procedural guidance altogether. Replicating the comparison in a second memory system would further establish whether the finding extends beyond SimpleMem.
- New-model null results cannot establish that the earlier effect was noise. The original backbones and judge are unavailable, and both change in the new experiments. The findings support limited transfer of the earlier result to current configurations, while leaving genuine model-dependent effects possible.
- There is no positive control demonstrating an effective intervention. Without genuinely useful retrieval procedures, unchanged accuracy could mean the planner ignores the injected block. Measure changes in retrieval plans, evidence recall, and instruction adherence; include task-relevant priors to establish that this injection mechanism can influence behavior.
- The comparison identifies selection effects, but not the proposed explanations for negative transfer. Relevance-versus-random cannot distinguish knowledge misalignment, domain mismatch, or adapter information loss, since these remain shared across both arms. Frame this as a narrower estimand, or add factorial interventions that vary bank relevance and adaptation independently.
Questions for authors
see weaknesses