- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RoutePrism to analyze how record arrival order affects agent memory construction and downstream answers. It examines source selection, compiled contexts, and answer changes, using repeated answers and evidence interventions to distinguish construction effects from sampling variation. Experiments across two datasets, multiple memory policies, and answer models illustrate how order changes affect answers through evidence retention and context changes.
Critical strengths
-
The paper goes beyond comparing final accuracy by examining source selection, compiled contexts, and answers separately, while using repeated answers to estimate sampling variation. This helps distinguish effects at different stages and avoids attributing every answer difference to memory construction.
-
The support-restoration experiments include non-support replacement and context-length controls, providing convincing evidence linking the loss of supporting records to answer performance. Replication across answer models and an additional dataset strengthens this observation.
Critical weaknesses
The main findings offer limited new insight. The largest accuracy effects occur with Focal and bounded-recent retention: reordering changes whether supporting evidence is retained, and restoring that evidence improves answers. Focal’s advantage under reversal can also be explained by the interaction between support positions and the retention rule. The authors validate these effects carefully and provide additional construction and context analyses, but I am not yet convinced that these results yield sufficiently significant, generalizable insights into memory mechanisms. My main concern is that the protocol adds diagnostic detail without clearly establishing a commensurate contribution to our understanding.
Questions for authors
-
I understand that the restoration experiments and retention-rule analyses establish how order affects answers, but several findings seem predictable from the policy definitions and the positions of supporting records. What do the authors consider the most important new insight from these experiments?
-
I’m interested in the MemoChat results. Could the authors provide a few paired examples showing how construction order changes the resulting context? Is supporting information lost during summarization or fragment selection, or does it remain intact while changes in wording or organization affect the answer?
-
The paper recommends comparing retention rules against task support before deployment. What additional help does RoutePrism provide over selecting a policy directly by validation accuracy? Could the authors give a concrete example of how the diagnosis informs a policy choice or modification, and evaluate it on questions not used to make that choice?
4 comments
The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.
AC
Thank you for the additional analysis and examples, but they still do not establish a clear, generalizable insight beyond the predictable effects of retention and context construction, so I maintain my rating of 2.
Unpredictable direction and a diagnosis-driven design change
We address the concern that our results are predictable from policy definitions. Two findings are not.
The same retention and retrieval rules favor opposite ingestion orders on different datasets. Both benchmarks use the same Focal compactor (cluster, keep last arrival) and the same BM25 top-three retrieval. The policy definition does not predict which direction wins, because that depends on where task-supporting records fall relative to the retention boundary in each dataset. We reproduce the relevant estimates from submitted Table 62.
| Benchmark | Questions | Forward support exposure | Reverse support exposure | Paired change |
|---|---|---|---|---|
| PersonaMem | 63 | 0.069 | 0.450 | +0.381 |
| LongMemEval-S | 470 | 0.146 | 0.106 | −0.039 |
PersonaMem's early supporting records are displaced by forward construction and recovered by reversal. LongMemEval-S shows the opposite pattern. The direction cannot be read off the policy; it requires tracing which evidence the construction actually exposes. This is the diagnostic gap RoutePrism fills.
Construction-order sensitivity persists after retention succeeds, and the diagnosis identified where to intervene. In MemoChat, 62 of 63 pairs selected the same source IDs (Table 5). Within the 45 pairs that also exposed the same complete records including all annotated support, the residual disagreement was 0.0488 [0.0045, 0.1231]. This sensitivity cannot be predicted from the retention rule, because retention succeeded. It arises from how the compiler arranges summaries, repetitions and positions around the retained records.
That observation directed us to change compilation rather than retention. We rendered each selected record in full, once, in source order, and tested the modification on eight users (24 questions) excluded from both diagnosis and selection.
| Candidate on held-out users | Luna accuracy | Luna D_res | Qwen3-8B accuracy | Qwen3-8B H_x |
|---|---|---|---|---|
| Original MemoChat compiler | 85.0% | 0.0933 | 47.5% | 27.1% |
| Each selected record once, in source order | 89.4% | 0.0076 | 37.8% | 0.0% |
| BM25 topic retrieval with original compiler | 90.3% | 0.0248 | 45.5% | 16.4% |
Validation accuracy selected the original compiler for Luna. The diagnostic selected the modification, reducing the held-out residual from 0.0933 to 0.0076. This is a design decision that differed between the two selection methods, with an independently measured stability improvement. The diagnostic provided information that accuracy selection alone did not.
Both paired accuracy-change intervals include zero. The transferable result is reduced sensitivity to construction order. Task accuracy remains an explicit selection criterion. We will add the cross-benchmark comparison and the held-out selection results to the revised manuscript.
Paired context analysis and cross-user compiler comparison
An assistant can retain every supporting passage and still change its answer when it rebuilds memory. In our MemoChat-style system, that observation identified a change to the context compiler that we could test on users excluded from the diagnosis.
What survives into the prompt
Early support explains Focal's reversal advantage (Appendix D.11). In MemoChat, 62 of 63 pairs had identical selected source IDs (Table 5). Our new analysis found that 17 of those 62 displayed different sets of complete records. The compiler takes the last two dialogs from each retrieved topic and appends recent dialogs. In 61 of all 63 pairs, both prompts contained every annotated supporting record in full. Across those pairs, cross-route disagreement exceeded same-prompt disagreement by 0.0375 (seven draws per route, Eq. 2).
Two paired contexts
Both examples use the seven-draw MemoChat trace in Appendix K. The codes below are PersonaMem query-ID prefixes.
ffe01d3c). Both prompts contained the same five full records. The self-help book passage appeared twice, starting at 9% and 78% of the forward context and at 36% and 51% of the reverse context. All seven forward draws selected the book recommendation. On the reverse prompt, two selected books and five selected journaling. The summaries, repetitions of other records and positions varied together.d3a4e999). The forward prompt included the stored passage about a community discussion. The reverse prompt retained a summary about an interest in medicine and omitted the discussion passage. Two of seven forward answers and six of seven reverse answers were correct.Choosing a modification on held-out users
We kept the source selection and rendered each selected record in full, once, in source order. We fixed three candidates before collecting answers and matched prompt lengths within each question and model. The diagnostic selector minimized the fraction of contexts that differed across builds, breaking ties by visible source overlap. It used no answers or support labels. The accuracy selector maximized mean accuracy over both builds. Both used the same 122 questions from 29 users. The 24 test questions came from eight other users, all outside the original diagnosis.
The diagnostic selected the source-order compiler for both models. Validation accuracy selected the original for Luna and the source-order compiler for Qwen. Luna's residual uses three draws per prompt. Qwen disagreement uses one greedy answer per route. The paired interval for the Luna residual change was [−0.1713, −0.0231]. Both accuracy-change intervals include zero. Qwen's held-out baseline is near chance level (47.5% on 24 questions from eight users), so its point estimates are noisy. On the 17 questions with distinct options, disagreement still fell for both models. The confidence interval for the Luna residual change then widened to include zero. We will add these paired examples and the selection comparison to the MemoChat results.