acceptodds
Under review as a conference paper at ICLR 2027

Daydreamer: Training and Evaluating Associative Memory Navigation for Persona-Grounded Generation

Abstract

Creative writing often draws on personal experience through association: one memory cues another, and fragments of experience are selectively recombined into new material. Yet existing benchmarks largely evaluate either the final creative artifact or one-shot memory retrieval, leaving this associative process itself unmeasured. We introduce DAYDREAMBENCH, a benchmark for studying episodic memory navigation in persona-grounded creative writing. It contains 46,959 episodic memories across 201 persona–corpus identities and 343 held-out writing tasks, with executable reference flows that connect literary works to plausible experiential memories. We further propose DAYDREAMER, a Writer–Walker framework in which a learned Walker sequentially searches, follows associations, retains useful episodes, and curates an ordered memory context before generation. The Walker is trained on executable trajectories followed by online GRPO with rewards for writing utility, reference recovery, and navigation progress. Across two Writer backbones, Daydreamer produces substantially broader memory discovery and better associative-flow alignment than untrained and retrieval-based baselines, while achieving the highest overall writing scores. The gains are most pronounced in persona grounding and creative transformation, suggesting that effective memory-augmented creative generation depends not only on retrieving relevant experiences, but on learning how to navigate and organize them.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
creative writing, large language models, memory navigation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_iJFDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DAYDREAMBENCH and DAYDREAMER. The benchmark connects 201 author personas with synthetic episodic MemoryBanks containing 46,959 first-person memories, including flows of 3–6 scenes reverse-constructed from source works. The memories are presented as plausible synthetic writing resources. The test set contains 343 tasks with personas held out from training and validation. DAYDREAMER separates a Writer from a Qwen3.5-2B Walker that navigates the MemoryBank through explicit actions. The Walker receives SFT on 470 trajectories followed by GRPO using writing-outcome, reference-based navigation, process, and protocol rewards; the direct incremental surface-recall reward is disabled in the reported run (Appendix B.3). With the Qwen writer, training increases surface recall from 0.395 to 0.508 and overall judge score from 76.89 to 77.75; the GPT-6 Luna writer's overall score increases from 82.08 to 82.89. Appendix F provides paired tests for navigation metrics, with the most consistent improvements in memory discovery and smaller, comparator-dependent improvements in final selection.

Critical strengths
  1. A useful formulation of sequential memory use. The separation of navigation from writing enables executable trajectory supervision and intermediate measurements of discovery, retention, and selection. This makes it possible to examine where a memory-augmented system succeeds or fails beyond the final generated text. The persona-disjoint evaluation also provides evidence of generalization to held-out personas within the constructed benchmark.
  2. Navigation improvements are evaluated with appropriate qualifications. Appendix F reports significance tests and bootstrap intervals. Keep Recall is not significantly higher than the base Walker (Holm-adjusted p = 0.7201), but improves over retrieval and Graph-PPR baselines. Selected F1 improves over the base Walker and Graph-PPR, with positive bootstrap intervals; against RAG/RAG+Reranking, the favorable sign test accompanies an interval that slightly crosses zero. Thus, the evidence supports both discovery and some final-selection improvements, with stronger consistency for discovery.
Critical weaknesses
  1. Independent validation of the reference construction and writing assessment is limited. The synthetic reference flows define several training rewards and navigation metrics. This supports measuring recovery of the constructed references, but does not establish that those references adequately represent the range of useful associations. The evaluation judge prompt in Appendix C, Table 11 also supplies reference memories and uses them when assessing persona grounding and creative transformation, so the writing scores are not fully independent of this construction. Useful non-reference memories could consequently be undervalued. Alignment of training and evaluation criteria does not make improvements on held-out personas automatic, but stronger independent validation is needed to interpret them as improvements in associative usefulness or creative writing quality.
  2. The incremental writing benefit of learned navigation remains uncertain. Overall writing-score gains over RAG+Reranking are 0.73 points for Qwen3.6-27B (77.75 vs. 77.02) and 0.50 points for GPT-6 Luna (82.89 vs. 82.39), without paired uncertainty estimates for these differences or human evaluation of the generated writing. Overlap between judge and generation model families warrants a sensitivity analysis, although it does not by itself demonstrate self-preference. The larger persona-grounding gain over Persona-only jointly reflects access to memories and the navigation strategy; it cannot isolate the contribution of learned navigation. The relevant incremental comparisons are against memory-augmented retrieval and the base Walker.
Questions for authors
  1. Can you provide paired confidence intervals and significance tests for writing-quality differences against RAG+Reranking and the base Walker, separately for each Writer?
  2. Can independent human raters evaluate generated writing without being required to reward agreement with the constructed reference flow? A useful accompanying check would assess the plausibility and usefulness of both reference and non-reference memory selections.
  3. How stable are the writing rankings when reference memories are omitted from the judging prompt, and when judges sharing a model family with the Writer or dataset generator are excluded?
  4. How sensitive are navigation results to alternative plausible reference flows or a different construction model? The relevant target is useful synthetic writing resources, consistent with the paper's stated scope.
  5. Since the retrieval arms share a base-Walker selector, can you add a direct RAG-to-Writer baseline to isolate the contribution of that shared selection stage?
3 comments
AuthorsOpenReview ↗

Response to Reviewer iJFD (1)

R1(W1, Q2, Q3). Reference-independent validation of writing quality and memory usefulness

We would first like to clarify an important distinction regarding the term “reference memories” in our evaluation prompt.

The memories provided to the writing Judge are not the gold reference-flow memories used to construct the training rewards and navigation metrics. Rather, reference memories are the memories actually retrieved and selected by each method and supplied to the Writer as writing references. The Judge does not observe the gold reference flow, gold memory set, or whether a selected memory belongs to that set. It only observes the memories available to the Writer and evaluates whether the generated text meaningfully grounds in and creatively transforms those memories.

Therefore, a memory is not penalized simply because it falls outside the constructed gold reference flow, since the Judge has no access to that membership information. A non-gold memory discovered by Daydreamer can receive full credit if it provides useful material for the resulting writing.

Following your suggestion, we conducted an additional human evaluation during the rebuttal period. We recruited 40 participants with an interest in literature and randomly sampled 50 writing tasks from the poetry portion of the test set to form the evaluation pool. Each participant was randomly assigned ten tasks. For each task, the participant compared anonymized outputs from four systems:

(1) Persona-only; (2) RAG + Reranking; (3) Base Walker; and (4) Daydreamer

Participants were not shown the method identity, retrieved memories, gold memories, or reference flows. They were asked to select the overall best poem based on four aspects of literary quality: language and imagery, poetic technique, originality, and emotional/aesthetic impact. They could also select “Tie / difficult to distinguish” when no output was clearly preferable.

The aggregate preferences are:

SystemHuman preference
Persona-only20.9%
RAG + Reranking22.6%
Base Walker22.1%
Daydreamer24.9%
Tie / difficult to distinguish9.5%

Daydreamer receives the highest human preference rate (24.9%) among the four systems. Importantly, this evaluation is completely independent of the constructed reference flows: annotators judge only the final writing and are never instructed to reward agreement with any reference memory or reference trajectory.

R2.(W2, Q1, Q3) Statistical reliability and judge-family sensitivity

We performed an additional reference-free pairwise evaluation over all 343 test writing tasks, comparing:

  1. Daydreamer vs. Base Walker; and
  2. Daydreamer vs. RAG + Reranking.

For each task, the two generated texts were anonymized and randomly assigned to positions A/B. Importantly, the evaluator was shown only the two generated texts: it did not observe the method identity, persona, retrieved memories, gold reference memories, reference flow, or navigation trajectory. The evaluator judged only the literary quality of the outputs and returned A, B, or Tie.

We evaluated the outputs using four independent model judges: Qwen, GPT-6 Luna, DeepSeek, and GLM. The resulting macro-persona preference rates are:

Judge settingDD vs. Base WalkerDD vs. RAG + Reranking
Average over all four judges52.4%53.1%
Excluding GPT-6 Luna53.5%53.5%

The preference for Daydreamer therefore persists when GPT-6 Luna is excluded, addressing the concern about overlap between the Writer and Judge model families. Across the test tasks, the paired comparisons remain statistically significant after Holm correction, with one-sided adjusted p-values of: (1) Daydreamer vs. Base Walker: p = 0.0439; (2) Daydreamer vs. RAG + Reranking: p = 0.0424

AuthorsOpenReview ↗

Response to Reviewer iJFD (2)

R3(Q4). Robustness to changes in the associative memory structure

Our original benchmark already adopts a persona-disjoint split, ensuring that training and test personas do not overlap. The evaluation therefore involves previously unseen personas and their corresponding MemoryBanks, providing initial evidence that the learned policy transfers across persona-specific memory environments and is robust to changes in the associative memory structure.

Secondly, to test whether the learned Walker depends on the particular associative structure present during training, we conducted an additional memory-graph perturbation experiment, without any additional training. Specifically, we perturb the memory traversal graph through semantically constrained edge rewiring:

  • Rewired concepts must be type-compatible with the original concepts, previously unconnected to the corresponding episode, and have an embedding cosine similarity of at least 0.70 to the original concepts.
  • We preserve each episode’s degree and constrain changes in concept-node degrees to avoid unrealistic graph corruption. Episode texts, memory IDs, embeddings, personas, writing tasks, Memory Requests, gold reference flows, and schema nodes remain unchanged.

We then evaluate 10% and 30% edge rewiring and all graph-aware methods operate on the same perturbed graph. We evaluate 20 held-out personas from our test split (randomly select 10 from PoetryDB and 10 from Gutenberg), covering 164 writing tasks, and compare four methods: trained Daydreamer, Base Walker, RAG + Reranking, and Graph-PPR.

Results. The following table reports persona-level macro-averaged Surface Recall. Perturbed results are averaged over three random seeds, with standard deviations shown.

MethodOriginal Graph10% Rewiring30% Rewiring
Daydreamer0.4810.457 ± 0.0240.469 ± 0.014
Base Walker0.3500.344 ± 0.0070.344 ± 0.004
RAG + Reranking0.3950.394 ± 0.0000.394 ± 0.001
Graph-PPR0.3650.360 ± 0.0030.344 ± 0.005

Daydreamer consistently outperforms all baselines even when the underlying associative graph is modified. Notably, under 30% edge rewiring, it achieves a Surface Recall of 0.469, compared with 0.344 for Base Walker and 0.394 for RAG + Reranking. Since the Walker is not retrained, these results suggest that the learned policy remains effective when navigating modified memory structures.

Together with our persona-disjoint evaluation, these findings support our central claim that Daydreamer learns a transferable, observation-conditioned policy for sequential associative memory navigation, rather than a strategy tied to a particular memory graph.

R4(Q5). Direct RAG baseline and the contribution of the shared selector

We would like to explain that our original motivation for applying the same selector to the retrieval baselines was to control for the final selection stage: without this control, differences between RAG and Daydreamer could conflate navigation/retrieval quality with the effect of an additional selector. Nevertheless, following your suggestion, we added a Direct RAG baseline in which the dense-retrieval Top-K memories are passed directly to the Writer without any Walker-based selection. We set (k=7) to match the average number of memories retained by the selector across test tasks, thereby avoiding an advantage from simply providing the Writer with more retrieved memories.

The results are:

MethodSurface RecallKeep RecallSelected F1Flow Align.Overall
Direct RAG39.628.419.3—79.68
RAG + Reranking39.725.919.8—82.39
Daydreamer52.232.022.619.482.89

First, the comparison shows that the shared selection stage itself improves downstream writing quality. This supports our original motivation for controlling the selector when comparing retrieval methods. Second, even after controlling for this shared selection stage, Daydreamer achieves higher navigation coverage and the final writing score.

AuthorsOpenReview ↗

Revised Manuscript and Further Discussion

Thank you for your suggestions. We have now uploaded the revised manuscript, incorporating additional experiments, analyses, and clarifications to address the reviewers' concerns. We sincerely invite you to review the updated manuscript and welcome any further questions or discussion.

Reviewer_mHv9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Daydreamer, an Associative Memory Navigation framework. Rather than enabling LLMs to retrieve relevant memories in a single shot via RAG, it mimics human recollection of experiences through multi-round searching, association, filtering and organization of memories to supply materials for personalized creative writing. A Walker is trained via SFT + GRPO to autonomously explore and structure memories.

Critical strengths
  1. The authors reframe conventional static Memory Retrieval as Sequential Memory Navigation defined by states, actions, observations and termination conditions.
  2. The authors reverse-engineer associable episodic memory graphs from existing literary works and replay them as executable trajectories, yielding training and evaluation signals for multi-step memory navigation. This perspective is original and innovative.
Critical weaknesses

My primary concern lies in the generalization of the Daydreamer framework. The reference memory chains in DaydreamBench are reverse-generated from target literary works, and the Memory Request construction process in the appendix also draws directly on the target works. This means the memory search requests at test time may already contain information originating from the target works — information that would not necessarily be available from real-world writing tasks and Personas. While this construction scheme provides clear supervision signals, it risks making the benchmark primarily measure the model’s ability to recover pre-constructed memory chains, rather than its capacity for memory navigation and creative utilization in open, realistic settings. Further validation is needed to verify whether the method retains its advantages when evaluated without search requests and reference conditions reverse-engineered from target works.

Questions for authors
  1. Regarding the benchmark’s dependency on target information: in practical inference scenarios, the Writer typically only has access to the Persona and the Writing Task. Were the fixed Memory Requests used in the experiments directly generated from the target works? If Memory Requests are constructed solely from the Persona and Writing Task, can Daydreamer still deliver comparable improvements in navigation and generation quality?
  2. Intuitively, Daydreamer shares conceptual common ground with world model training: both formulate tasks as sequential decision-making processes consisting of states, actions and state transitions, and train models for multi-step decision-making using interactive trajectories. Could the authors clarify whether the Walker learns a transferable internal representation of associative memory structure, or whether its gains stem mainly from learning more effective tool-use strategies within a fixed retrieval environment? An additional experiment assessing its ability to predict association outcomes or generalize to modified memory graphs and unseen association patterns would help disentangle these two possibilities.
2 comments
AuthorsOpenReview ↗

Response to Reviewer mHv9 (1)

R1 (W1 + Q1). Generalization Beyond Target-Derived Memory Requests

Response: To address your concern, we conducted an additional experiment to examine whether Daydreamer remains beneficial when writing tasks and memory requests are constructed independently of the target literary works.

We introduce three mundane, externally specified inspiration seeds (“window,” “orange,” and “coat”), which are independent of all target works and shared across personas. For each of the 40 held-out personas, we construct three seed-based writing tasks, resulting in 120 writing instances.

Crucially, the request generator has no access to the target work, gold memory chain, reference memories, original benchmark request, or evaluation labels. This setting allows different personas to interpret the same external inspiration through their own perspectives while eliminating the direct dependence on reverse-engineered target works. We compare three conditions using the same GPT-6 Luna Writer: Daydreamer; Base Walker; and Persona-only.

Reference-Free Evaluation: We conduct blinded pairwise comparisons between Daydreamer and each baseline. Candidate positions are randomized and balanced across genres. The judges are not provided with memory traces, persona descriptions, target works, or reference memories. Each judge provides (i) an anonymous pairwise preference and (ii) independent overall literary-quality scores on a 0–10 scale for both responses.

We aggregate results from four independent judges used in the paper (GPT, Qwen, GLM, and DeepSeek) and report Daydreamer’s win rate and mean score improvement over each baseline.

ComparisonDaydreamer Win RateMean Score Improvement
Daydreamer vs. Base Walker51.81%+2.9%
Daydreamer vs. Persona-only50.83%+5.0%

From the table we can see that Daydreamer achieves higher writing-quality scores even when the writing tasks and memory requests are independent of the target works. Importantly, Daydreamer treats retrieved memories as optional inspiration rather than mandatory content. This allows the Writer to draw on useful experiences when relevant, while avoiding the forced use of unrelated memories. This flexibility is particularly important for open-ended writing, where an external prompt may not closely match the persona's past experiences.

Despite this challenge, Daydreamer still improves the overall writing scores, suggesting that episodic memories can provide valuable creative support even without target-derived guidance. The benefits are greater when the writing task connects more naturally to the persona's memories.

AuthorsOpenReview ↗

Response to Reviewer mHv9 (2)

R1(W1 + Q2). Transferability of the Learned Navigation Policy

We appreciate your insightful connection between Daydreamer and world-model learning. Our motivation is to investigate whether models can learn generalizable associative navigation strategies from the relationships between literary works and possible experiential memories. Specifically, we aim to learn how skilled writers connect different experiences and transform them into creative inspiration, rather than simply retrieving relevant memories.

First, our original benchmark already adopts a persona-disjoint split, ensuring that training and test personas do not overlap. The evaluation therefore involves unseen personas and their corresponding MemoryBanks, providing initial evidence that the learned navigation policy can transfer across different memory environments.

Second, we conduct an additional memory-graph perturbation experiment to further examine whether Walker relies on fixed associative structures. Without retraining the model, we modify the memory graph through semantically constrained edge rewiring:

  • Rewired concepts must be type-compatible with the original concepts, previously unconnected to the corresponding episode, and have an embedding cosine similarity of at least 0.70 to the original concepts.
  • We preserve each episode’s degree and constrain changes in concept-node degrees to avoid unrealistic graph corruption. Episode texts, memory IDs, embeddings, personas, writing tasks, Memory Requests, gold reference flows, and schema nodes remain unchanged.

We then evaluate 10% and 30% edge rewiring and all graph-aware methods operate on the same perturbed graph. We evaluate 20 held-out personas from our test split (randomly select 10 from PoetryDB and 10 from Gutenberg), covering 164 writing tasks, and compare four methods: trained Daydreamer, Base Walker, RAG + Reranking, and Graph-PPR.

Results. The following table reports persona-level macro-averaged Surface Recall. Perturbed results are averaged over three random seeds, with standard deviations shown.

MethodOriginal Graph10% Rewiring30% Rewiring
Daydreamer0.4810.457 ± 0.0240.469 ± 0.014
Base Walker0.3500.344 ± 0.0070.344 ± 0.004
RAG + Reranking0.3950.394 ± 0.0000.394 ± 0.001
Graph-PPR0.3650.360 ± 0.0030.344 ± 0.005

Daydreamer consistently outperforms all baselines even when the underlying associative graph is modified. Notably, under 30% edge rewiring, it achieves a Surface Recall of 0.469, compared with 0.344 for Base Walker and 0.394 for RAG + Reranking. Since the Walker is not retrained, these results suggest that the learned policy remains effective when navigating modified memory structures.

Together with our persona-disjoint evaluation, these findings support our central claim that Daydreamer learns a transferable, observation-conditioned policy for sequential associative memory navigation, rather than a strategy tied to a particular memory graph.

Comments on the paper

16 comments
AuthorsOpenReview ↗

Response to Reviewer rSeF (1)

R3 (W1 + Q1 + Q2). Learning and Evaluating Associations Beyond Reference Flows

Thank you for your suggestions. Our goal is to learn how skilled writers connect different experiences and turn them into creative inspiration. We use associative paths drawn from existing literary works as training examples, but do not treat them as the only valid associations.

(1) Reward Design and Non-Reference Associations.

Our training combines reference-based navigation rewards with an outcome reward. The F1 and flow rewards encourage the Walker to recover known, meaningful associations, while the outcome reward evaluates whether the selected memories improve the final writing. Non-gold memories may receive less navigation reward, but they can still contribute through the outcome reward if they help the writing. We acknowledge that reference-based rewards introduce a bias toward the constructed flows, but recovering these flows is not our only objective.

We also want to clarify that the "reference memories" shown to our writing Judge are the memories actually selected by each method and supplied to the Writer, not the hidden gold reference set used for navigation supervision. The Judge does not observe the gold memory set, reference flow, or whether a selected memory is gold. It evaluates how well the writing uses the supplied memories, including non-gold ones.

To further examine whether Daydreamer can find useful associations beyond the constructed references, we conducted the following experiments.

(2) Non-Gold Memory Masking.

We conducted a new memory-masking experiment, randomly sampling half of the poetry tasks and half of the short-story tasks. Using the original Daydreamer trajectories, we kept the same persona, writing task, and memory request, but changed the memories passed to the Writer:

  • Full: All memories selected by the Walker.

  • Gold-only: Only selected memories belonging to the gold reference set.

  • Non-gold-only: Only selected memories outside the gold reference set.

We did not rerun the Walker. Instead, we regenerated the responses with GPT-6 Luna using the masked memory sets and evaluated them with the same four LLM judges.

Memory conditionOverall
Full84.67
Gold-only84.07
Non-gold-only83.89

Interestingly, Non-gold-only contains no gold reference memories by construction, yet achieves an Overall score of 83.89, only 0.18 points below Gold-only (84.07). Full memory performs best (84.67), suggesting that gold and non-gold memories may provide complementary writing material. These results suggest that non-gold memories are not just alternatives to the reference memories; they may also complement them, providing additional inspiration that helps the Writer develop the original associations further.

(3) Writing Tasks Independent of Target Literary Works.

We also tested whether Daydreamer remains useful when writing tasks are constructed independently of the target literary works.

We introduced three external inspiration seeds ("window," "orange," and "coat"), shared across all 40 held-out personas, resulting in 120 new writing tasks. The request generator had no access to the target works, gold memory chains, reference memories, original benchmark requests, or evaluation labels.

We compared Daydreamer, Base Walker, and Persona-only using the same GPT-6 Luna Writer. For evaluation, we conducted blinded pairwise comparisons with four LLM judges (GPT, Qwen, GLM, and DeepSeek). Candidate positions were randomized, and judges saw neither the method identities nor any reference memories or navigation traces. Each judge provided a pairwise preference and independent literary-quality scores on a 0–10 scale.

ComparisonDaydreamer Win RateMean Score Improvement
Daydreamer vs. Base Walker51.81%+2.9%
Daydreamer vs. Persona-only50.83%+5.0%

The results show that Daydreamer achieves higher writing-quality scores even when the writing tasks and memory requests are independent of the target works. Importantly, Daydreamer treats retrieved memories as optional inspiration rather than mandatory content. This allows the Writer to draw on useful experiences when relevant, while avoiding the forced use of unrelated memories. This flexibility is particularly important for open-ended writing, where an external prompt may not closely match the persona's past experiences. Despite this challenge, Daydreamer still improves the overall writing scores, suggesting that episodic memories can provide valuable creative support even without target-derived guidance. The benefits are greater when the writing task connects more naturally to the persona's memories.

AuthorsOpenReview ↗

Response to Reviewer rSeF (2)

R3 (W1 + Q1 + Q2). Learning and Evaluating Associations Beyond Reference Flows

(continued from Response to Reviewer rSeF (1))

(4) Reference-Free Evaluation of Writing Quality.

We further conducted both LLM and human evaluations that are independent of the constructed reference memories.

LLM evaluation. We performed blinded pairwise comparisons across all 343 test tasks, comparing Daydreamer against Base Walker and RAG + Reranking. Each pair of generated texts was anonymized and randomly assigned to positions A/B. The judges saw only the two texts, without method identities, personas, retrieved memories, target works, or reference flows, and selected A, B, or Tie.

We used four independent judges (Qwen, GPT-6 Luna, DeepSeek, and GLM). The macro-persona preference rates are:

Judge settingDD vs. Base WalkerDD vs. RAG + Reranking
All four judges52.4%53.1%
Excluding GPT-6 Luna53.5%53.5%

The preference for Daydreamer remains when GPT-6 Luna is excluded. Both comparisons are statistically significant after Holm correction, with one-sided adjusted p-values of 0.0439 and 0.0424, respectively.

Human evaluation: We recruited 40 participants interested in literature and randomly sampled 50 poetry tasks from the test set. Each participant evaluated ten tasks, comparing anonymized outputs from Persona-only, RAG + Reranking, Base Walker, and Daydreamer.

Participants selected the best poem based on language and imagery, poetic technique, originality, and emotional/aesthetic impact. They could also select "Tie / difficult to distinguish." They were not shown any method identities, retrieved memories, or reference flows.

SystemHuman Preference
Persona-only20.9%
RAG + Reranking22.6%
Base Walker22.1%
Daydreamer24.9%
Tie / difficult to distinguish9.5%

Daydreamer receives the highest human preference rate. Importantly, neither the human participants nor the reference-free LLM judges were asked to reward agreement with any gold memories or reference trajectories.

Taken together, these experiments show that Daydreamer's writing benefits are not limited to recovering the constructed reference memories. While reference-based rewards may still favor known associations, the Walker can also find non-gold memories that support creative writing.

AuthorsOpenReview ↗

Response to Reviewer rSeF (3)

R3 (W2 + Q3). What Does the Walker Learn Beyond Adaptive Retrieval?

We considered this limitation in our design. Rather than relying only on the original query, Daydreamer uses associative spread after Keep, where retained memories become new retrieval cues.

sispread=(1−μ)vi⊤vˉt+μvi⊤E(r)s_i^{\mathrm{spread}} = (1-\mu)v_i^\top\bar v_t +\mu v_i^\top E(r)

We set �0��0�, so the spread score combines the retained-memory context (75%) with the original request (25%), allowing the search to gradually shift toward related experiences that may not be retrieved directly.

We conducted two additional experiments to better understand what the Walker learns.

(1) Ablation of Navigation Operations.

To separate the contributions of adaptive querying, associative spread, and concept traversal, we evaluated the same trained Walker with matched action budgets and MemoryBanks under three settings: Fixed-query replaces generated queries with the fixed task-plus-request query; No-spread disables retrieval expansion after Keep; and No-follow disables explicit concept-link traversal.

MethodSurface RecallKeep RecallSelected F1Flow Alignment
Full0.51740.33060.22570.1993
Fixed-query0.43300.28180.20160.1635
No-spread0.35350.30750.23480.1741
No-follow0.53530.32390.22340.1951

Findings. The full model achieves the highest Flow Alignment (0.1993), compared with 0.1635 for Fixed-query and 0.1741 for No-spread. This suggests that both adaptive querying and memory-conditioned associative spread help the Walker recover meaningful multi-step associations. Interestingly, No-spread achieves a slightly higher Selected F1 (0.2348 vs. 0.2257), but substantially lower Flow Alignment. This highlights an important distinction: recovering the right memories is not the same as recovering the associative paths connecting them. Our results suggest that the learned navigation behavior mainly comes from adaptive querying and associative spread, rather than explicit graph traversal.

(2) Robustness to Changes in the Memory Graph.

First, our original benchmark already adopts a persona-disjoint split, ensuring that training and test personas do not overlap. The evaluation therefore involves unseen personas and their corresponding MemoryBanks, providing initial evidence that the learned navigation policy can transfer across different memory environments.

Second, we conducted an additional memory-graph perturbation experiment to further examine whether Walker relies on fixed associative structures. Without retraining the model, we modify the memory graph through semantically constrained edge rewiring:

  • (1) Rewired concepts must be type-compatible with the original concepts, previously unconnected to the corresponding episode, and have an embedding cosine similarity of at least 0.70 to the original concepts.
  • (2) We preserve each episode’s degree and constrain changes in concept-node degrees to avoid unrealistic graph corruption. Episode texts, memory IDs, embeddings, personas, writing tasks, Memory Requests, gold reference flows, and schema nodes remain unchanged.

We then evaluate 10% and 30% edge rewiring and all graph-aware methods operate on the same perturbed graph. We evaluate 20 held-out personas from our test split (randomly select 10 from PoetryDB and 10 from Gutenberg), covering 164 writing tasks, and compare four methods: trained Daydreamer, Base Walker, RAG + Reranking, and Graph-PPR.

Results. The following table reports persona-level macro-averaged Surface Recall. Perturbed results are averaged over three random seeds, with standard deviations shown.

MethodOriginal Graph10% Rewiring30% Rewiring
Daydreamer0.4810.457 ± 0.0240.469 ± 0.014
Base Walker0.3500.344 ± 0.0070.344 ± 0.004
RAG + Reranking0.3950.394 ± 0.0000.394 ± 0.001
Graph-PPR0.3650.360 ± 0.0030.344 ± 0.005

With 30% rewiring, Daydreamer still achieves 0.469 Surface Recall, outperforming Base Walker (0.344) and RAG + Reranking (0.394). This shows that its performance advantage persists when the predefined graph is modified, even without retraining.

Taken together, these results clarify what the Walker learns: not new associative representations, but a sequential search strategy that uses retrieved experiences to guide further memory exploration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.