acceptodds
Under review as a conference paper at ICLR 2027

RoutePrism: Tracing Construction Order Effects in Agent Memory

Abstract

Processing the same records in a different order can discard different evidence, yet endpoint accuracy alone cannot reveal what changed or whether it mattered. We introduce RoutePrism, a diagnostic protocol that builds memory twice from the same source pool in two processing orders, then traces which sources, compiled contexts, and answers differ. Because record content, timestamps, policy, and the answer model all stay fixed, any observed difference is localized to the memory construction step. A matched four-condition intervention tests whether a record displaced by reordering actually carried task-relevant evidence: restoring that single record recovers over 60 percentage points of lost accuracy, while substituting a non-supporting record of equal length does not. We evaluate the protocol on PersonaMem-32K (63 primary queries, 29 users) and 470 LongMemEval-S questions with histories spanning 38 to 62 sessions, replicating the core intervention across five answer models. Survivor selection, defined as the choice of which record a cluster retains, drives most source-level changes, while different memory policies (compaction, bounded recency, MemoChat-style summarization, A-MEM) produce distinct failure signatures at the source, context, and metadata layers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x7tTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RoutePrism to analyze how record arrival order affects agent memory construction and downstream answers. It examines source selection, compiled contexts, and answer changes, using repeated answers and evidence interventions to distinguish construction effects from sampling variation. Experiments across two datasets, multiple memory policies, and answer models illustrate how order changes affect answers through evidence retention and context changes.

Critical strengths
  1. The paper goes beyond comparing final accuracy by examining source selection, compiled contexts, and answers separately, while using repeated answers to estimate sampling variation. This helps distinguish effects at different stages and avoids attributing every answer difference to memory construction.

  2. The support-restoration experiments include non-support replacement and context-length controls, providing convincing evidence linking the loss of supporting records to answer performance. Replication across answer models and an additional dataset strengthens this observation.

Critical weaknesses

The main findings offer limited new insight. The largest accuracy effects occur with Focal and bounded-recent retention: reordering changes whether supporting evidence is retained, and restoring that evidence improves answers. Focal’s advantage under reversal can also be explained by the interaction between support positions and the retention rule. The authors validate these effects carefully and provide additional construction and context analyses, but I am not yet convinced that these results yield sufficiently significant, generalizable insights into memory mechanisms. My main concern is that the protocol adds diagnostic detail without clearly establishing a commensurate contribution to our understanding.

Questions for authors
  1. I understand that the restoration experiments and retention-rule analyses establish how order affects answers, but several findings seem predictable from the policy definitions and the positions of supporting records. What do the authors consider the most important new insight from these experiments?

  2. I’m interested in the MemoChat results. Could the authors provide a few paired examples showing how construction order changes the resulting context? Is supporting information lost during summarization or fragment selection, or does it remain intact while changes in wording or organization affect the answer?

  3. The paper recommends comparing retention rules against task support before deployment. What additional help does RoutePrism provide over selecting a policy directly by validation accuracy? Could the authors give a concrete example of how the diagnosis informs a policy choice or modification, and evaluate it on questions not used to make that choice?

4 comments
AuthorsOpenReview ↗

Paired context analysis and cross-user compiler comparison

An assistant can retain every supporting passage and still change its answer when it rebuilds memory. In our MemoChat-style system, that observation identified a change to the context compiler that we could test on users excluded from the diagnosis.

What survives into the prompt

Early support explains Focal's reversal advantage (Appendix D.11). In MemoChat, 62 of 63 pairs had identical selected source IDs (Table 5). Our new analysis found that 17 of those 62 displayed different sets of complete records. The compiler takes the last two dialogs from each retrieved topic and appends recent dialogs. In 61 of all 63 pairs, both prompts contained every annotated supporting record in full. Across those pairs, cross-route disagreement exceeded same-prompt disagreement by 0.0375 (seven draws per route, Eq. 2).

Two paired contexts

Both examples use the seven-draw MemoChat trace in Appendix K. The codes below are PersonaMem query-ID prefixes.

  • Advice about thought patterns (ffe01d3c). Both prompts contained the same five full records. The self-help book passage appeared twice, starting at 9% and 78% of the forward context and at 36% and 51% of the reverse context. All seven forward draws selected the book recommendation. On the reverse prompt, two selected books and five selected journaling. The summaries, repetitions of other records and positions varied together.
  • A discussion of chronic illness (d3a4e999). The forward prompt included the stored passage about a community discussion. The reverse prompt retained a summary about an interest in medicine and omitted the discussion passage. Two of seven forward answers and six of seven reverse answers were correct.

Choosing a modification on held-out users

We kept the source selection and rendered each selected record in full, once, in source order. We fixed three candidates before collecting answers and matched prompt lengths within each question and model. The diagnostic selector minimized the fraction of contexts that differed across builds, breaking ties by visible source overlap. It used no answers or support labels. The accuracy selector maximized mean accuracy over both builds. Both used the same 122 questions from 29 users. The 24 test questions came from eight other users, all outside the original diagnosis.

Candidate on the held-out usersLuna accuracyLuna residual disagreementQwen3-8B accuracyQwen3-8B disagreement
Original MemoChat compiler85.0%0.093347.5%27.1%
Render each selected source once in source order89.4%0.007637.8%0.0%
BM25 topic retrieval with original compiler90.3%0.024845.5%16.4%

The diagnostic selected the source-order compiler for both models. Validation accuracy selected the original for Luna and the source-order compiler for Qwen. Luna's residual uses three draws per prompt. Qwen disagreement uses one greedy answer per route. The paired interval for the Luna residual change was [−0.1713, −0.0231]. Both accuracy-change intervals include zero. Qwen's held-out baseline is near chance level (47.5% on 24 questions from eight users), so its point estimates are noisy. On the 17 questions with distinct options, disagreement still fell for both models. The confidence interval for the Luna residual change then widened to include zero. We will add these paired examples and the selection comparison to the MemoChat results.

The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.

AC

Thank you for the additional analysis and examples, but they still do not establish a clear, generalizable insight beyond the predictable effects of retention and context construction, so I maintain my rating of 2.

AuthorsOpenReview ↗

Unpredictable direction and a diagnosis-driven design change

We address the concern that our results are predictable from policy definitions. Two findings are not.

The same retention and retrieval rules favor opposite ingestion orders on different datasets. Both benchmarks use the same Focal compactor (cluster, keep last arrival) and the same BM25 top-three retrieval. The policy definition does not predict which direction wins, because that depends on where task-supporting records fall relative to the retention boundary in each dataset. We reproduce the relevant estimates from submitted Table 62.

BenchmarkQuestionsForward support exposureReverse support exposurePaired change
PersonaMem630.0690.450+0.381
LongMemEval-S4700.1460.106−0.039

PersonaMem's early supporting records are displaced by forward construction and recovered by reversal. LongMemEval-S shows the opposite pattern. The direction cannot be read off the policy; it requires tracing which evidence the construction actually exposes. This is the diagnostic gap RoutePrism fills.

Construction-order sensitivity persists after retention succeeds, and the diagnosis identified where to intervene. In MemoChat, 62 of 63 pairs selected the same source IDs (Table 5). Within the 45 pairs that also exposed the same complete records including all annotated support, the residual disagreement was 0.0488 [0.0045, 0.1231]. This sensitivity cannot be predicted from the retention rule, because retention succeeded. It arises from how the compiler arranges summaries, repetitions and positions around the retained records.

That observation directed us to change compilation rather than retention. We rendered each selected record in full, once, in source order, and tested the modification on eight users (24 questions) excluded from both diagnosis and selection.

Candidate on held-out usersLuna accuracyLuna D_resQwen3-8B accuracyQwen3-8B H_x
Original MemoChat compiler85.0%0.093347.5%27.1%
Each selected record once, in source order89.4%0.007637.8%0.0%
BM25 topic retrieval with original compiler90.3%0.024845.5%16.4%

Validation accuracy selected the original compiler for Luna. The diagnostic selected the modification, reducing the held-out residual from 0.0933 to 0.0076. This is a design decision that differed between the two selection methods, with an independently measured stability improvement. The diagnostic provided information that accuracy selection alone did not.

Both paired accuracy-change intervals include zero. The transferable result is reduced sensitivity to construction order. Task accuracy remains an explicit selection criterion. We will add the cross-benchmark comparison and the held-out selection results to the revised manuscript.

Reviewer_ijoiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces RoutePrism, a diagnostic protocol for studying how processing order affects agent memory. It constructs memory from the same records in forward and reversed orders while keeping record contents, timestamps, and policy settings fixed. The protocol tracks changes in retained sources, compiled contexts, and answers, with repeated answer draws used to distinguish construction effects from answer sampling. The authors then restore displaced supporting records and compare them with matched non-supporting replacements to test whether the lost evidence affects accuracy. Experiments on PersonaMem and LongMemEval-S cover several memory policies and answer models. The main findings are that survivor selection drives most source changes in the focal compactor, restoring missing support substantially recovers accuracy, and different policies exhibit changes at different observation layers.

Critical strengths

The support-restoration experiment includes a length-matched non-supporting replacement, which is a useful control for testing whether the displaced record’s content matters.

Critical weaknesses
  1. The problem has limited demonstrated practical significance, and the main conclusions add little insight. The main comparison reverses the processing order of an entire record pool while preserving timestamps, without establishing how representative this disruption is of practical ingestion. The strongest mechanism finding comes from a compactor explicitly designed to retain the last arrival in each cluster. Supporting records tend to appear early in the benchmark, so reversing the order predictably protects them. Showing that survivor selection drives this effect, and that restoring missing support improves fact recall, offers limited insight beyond the chosen policy and data layout. The additional policies broaden the evaluation, but the paper does not demonstrate that RoutePrism leads to substantive new memory-design decisions. Its recommendations on provenance tracking, logging, and survivor-rule auditing remain generic, with no demonstrated benefit from applying them.

  2. The restoration gains support a narrow conditional conclusion, while some reported “failures” lack demonstrated harm. The primary focal restoration analysis on PersonaMem uses 33 queries, and the LongMemEval restoration analysis uses 24 single-session questions. These are selected restoration subsets; the broader LongMemEval evaluation covers 470 questions. Selection requires a displaced supporting record and satisfaction of the matching conditions. Large recovery gains therefore establish the value of restoring known support in those cases, rather than the average impact of construction-order effects across the full workload. The broader PersonaMem “All eligible” analysis retains the structural selection requirements, so it cannot establish general representativeness. Separately, A-MEM exhibits metadata changes while its exposed sources and answer prompt remain unchanged, with no demonstrated downstream harm. Grouping these changes under “failure modes” overstates the evidence. Detecting additional internal differences does not establish that the protocol identifies additional consequential memory failures.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Compiler modification, held-out validation, and whole-set impact

Tracing MemoChat's records into the answer prompt led us to render each selected record once in source order. On users held out from diagnosis and policy selection, this change reduced answer disagreement under reordering for both models.

What the diagnosis changed

Table 5 shows why we investigated compilation. MemoChat selected the same source IDs in 62 of 63 pairs, yet the text sent to the answer model changed in all 63. Our new analysis traced the records into the final prompts. Both prompts still contained every annotated supporting record in 61 pairs. Even in that subset, disagreement across routes exceeded disagreement between repeated answers to the same prompt by 0.0375, using seven draws per route. This directed us to change how the retrieved records were presented.

With source selection fixed, the modified compiler preserved every complete record the original displayed across 292 builds. It exposed additional complete records in 52 builds. The test covers all 146 PersonaMem fact-recall questions, including 24 from eight held-out users.

QuestionsModelMean accuracy, original / modifiedDisagreement, original / modified
All 146Luna82.8% / 80.7%0.0732 / essentially zero
All 146Qwen3-8B60.1% / 59.1%17.5% / 1.5%
Held-out 24Luna85.0% / 89.4%0.0933 / 0.0076
Held-out 24Qwen3-8B47.5% / 37.8%27.1% / 0.0%

Luna's residual subtracts same-prompt disagreement. Qwen disagreement uses one greedy answer per route. Both models showed greater stability. The paired accuracy intervals include zero. Qwen's held-out baseline is near chance level (47.5% on 24 questions from eight users), so its point estimates are noisy.

What depends on the retention rule

Early support explains Focal's reversal advantage on PersonaMem (Appendix D.11). Under the half-swap, letting only survivor selection follow arrival order changed the exposed sources on 63 of 63 questions. Letting only clustering follow arrival order changed them on two (Table 4). This identifies the operation to inspect. Archived sessions can arrive for import in a different order from their event dates. Reversal tests sensitivity to that distinction. Measuring its frequency in deployment requires ingestion logs.

Conditional recovery and whole-set impact

Among the 33 eligible Focal questions, Luna accuracy was 21.0% with support absent, 88.1% with that record restored and 28.0% with a non-support replacement at the same prompt length and source count (Table 3). The expanded analysis keeps those structural criteria (78 Focal and 71 Recent-3 questions). The 24 LongMemEval restoration cases are single-session-user questions whose evidence session reached the prompt on exactly one route (Table 8). The separate Recent-8/BM25 comparison includes all 470 non-abstention questions, regardless of support exposure. Reverse import lowered accuracy by 12.6 percentage points for Luna and 11.5 for Qwen3-8B (Appendix O.6, Table 68).

For A-MEM, all 63 pairs exposed the same sources and the same answer prompt (Table 5). We will describe the observed metadata differences as internal variation and reserve task-harm language for measured answer degradation. We will also place the whole-set result beside the conditional restoration result and add the held-out compiler comparison to the MemoChat discussion.

The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.

AC

Reviewer_D4yWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is primarily a diagnostic for understanding how memory systems change based on their processing order. They use a few different orders (reversal, half-swap, odd-even) while keeping benchmark questions and ground truth answers fixed, and carefully control for cases where reordering should not change the correct answer. The results indicate that order can impact what is stored in memory as well as downstream task performance.

Critical strengths

S1 - Memory has become ubiquitous across interactive agents. It is a very relevant topic area for research and the way it is done through multiple rounds of compaction deserves additional study.

S2 - The paper goes to great lengths to control for order effects, albeit it is not clear if the specific orderings being studied are sufficiently motivated.

S3 - The paper is rich in empirical results, measuring different memory policies and using multiple benchmarks, as well introducing some of their own metrics.

Critical weaknesses

W1 - The paper does not sufficiently motivate the need for AI memory to be robust to arbitrary orderings. Especially for the chat domain, it's plausible that there are natural orderings. The paper does control for orderings that are roughly equivalent, i.e. the order of statements should not override each other. It also incidentally uncovers interesting edge cases, such as recent updates being more well represented. But does not go far enough to show a challenge set worth specifically optimizing for.

W2 - Similar to W1, the paper does not make clear the practical implications of its findings. This could be addressed via some "fix" that makes the memory systems more robust to order, and would be strengthened by showing a robust-to-order system simply generalizes better across held-out memory benchmarks.

W3 - The paper overclaims that accuracy is not enough. A deliberately constructed evaluation set may indeed prove sufficient, and the authors should more strongly justify why their introduced diagnostics are needed if we had such challenge sets in hand.

Questions for authors

Comment: In prompt-based listwise ranking, it is often the case that final accuracy is improved when the documents are presented in a natural order (e.g. ranked by an embedding) prior to running the reranker prompt. There may be a loose connection to processing memory as well.

2 comments
AuthorsOpenReview ↗

Causal decomposition of order sensitivity and cross-user compiler test

We address your concern about the practical relevance of import-order sensitivity with both a causal decomposition and a cross-user compiler test.

Under a half-swap, Focal changed the sources exposed to the answer model on all 63 questions, while its single-draw accuracy estimate moved by just 1.1 percentage points. Letting only survivor selection follow arrival order still changed the sources on all 63 questions. Letting only clustering follow arrival order changed them on two (Table 4). A challenge set measures the performance change. These interventions locate the construction step responsible for displacing evidence.

When import order matters

A conversation has an event order even when its archived sessions arrive in a different sequence during a rebuild. Graphiti's API, for example, accepts an occurrence time for conversational episodes and supports batch imports. Section 3.1 keeps dates and within-session turns fixed. On PersonaMem two independent readings had to agree that the full history supports the same answer under either arrangement. Reversal is a stress test. The half-swap changes which block arrives first while preserving order within each block. Estimating how often a deployed system encounters these schedules would require its ingestion logs. Graphiti episode documentation

Your ranking analogy corresponds to the control in Table 6. It permutes the display order of already constructed blocks while holding their text fixed. The intervention in Table 4 changes which records reach that display.

A modification tested on other users

In MemoChat the selected sources often agreed while the compiled text differed (Table 5). We therefore kept the source selection and rendered each selected record in full, once, in source order. Our new experiment compared this compiler, the original compiler and a BM25 retrieval variant with matched prompt lengths. In five-fold validation across users, the diagnostic selector gave Luna 80.7% accuracy and residual disagreement near zero. Validation-accuracy selection gave 81.7% accuracy and a residual of 0.0966. The residual subtracts same-prompt disagreement (Eq. 2). For Qwen3-8B the two selectors gave 59.1% and 57.3% accuracy, with observed disagreement of 1.5% and 9.7%, respectively.

We also reserved eight users (24 questions) who contributed to neither the original diagnosis nor selection among the candidates. For Luna, validation accuracy chose the original compiler and the diagnostic chose the modification. The held-out residual fell from 0.0933 to 0.0076. Mean accuracy went from 85.0% to 89.4%. For Qwen3-8B both selectors chose the modification. Relative to the original, observed disagreement fell from 27.1% to zero and mean accuracy went from 47.5% to 37.8%. Both paired accuracy intervals include zero.

The compiler transfer test is across PersonaMem users. The separate cross-benchmark evidence concerns the diagnostic protocol. Across all 470 non-abstention LongMemEval-S questions, reversal reduced Recent-8/BM25 accuracy by 12.6 points for Luna and 11.5 for Qwen3-8B (Appendix O.6, Table 68). We will clarify the distinction between detecting a performance change and locating its cause in the introduction, and add the held-out comparison to the MemoChat results.

The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.