acceptodds
Under review as a conference paper at ICLR 2027

Librarian Agents: Anticipatory Filesystem Construction via Reinforcement Learning

Abstract

Reasoning over long document contexts remains a challenge for large language models (LLMs). Standard retrieval methods are inherently reactive, bottlenecking agents with complex, cross-document reasoning at inference time. Existing approaches that pre-compute corpus structure, such as knowledge graphs, often fall short due to their reliance on rigid heuristics such as identifying interdocument relationships via named entity recognition. We propose an anticipatory approach to memory: proactively reorganizing unstructured document corpora to efficiently serve diverse future queries. This pre-computation synthesizes and restructures information within documents in advance, significantly improving an answering agent's downstream performance. While agentic scaffolds (e.g., Claude Code) are heavily trained to navigate and extend filesystems while solving the task currently in front of them, they do not pre-construct one in anticipation of future queries that our approach aims to do. We introduce Librarian Agents, a novel framework that restructures unstructured corpora into file-based memories arranged around the operations that LLM agents already perform well, such as reading summaries and following links. Librarian Agents is trained to maximize downstream success rate via end-to-end reinforcement learning. Empirically, Librarian Agents improves accuracy over the best performing baseline by 33 points on BrowseComp+ and additionally generalizes to out-of-distribution corpora such as BRIGHT with a 31 point increase on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, reinforcement learning in agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iHq6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Librarian Agents, a framework for anticipatory memory construction over long document corpora. Instead of performing retrieval independently for each query, a builder agent proactively reorganizes the corpus into a hierarchical filesystem containing synthetic summaries, merged artifacts, and provenance links to the original documents. A separate answering agent then navigates this filesystem to answer future queries.

Critical strengths
  1. A new formulation of anticipatory memory construction. The paper identifies an important limitation of reactive retrieval systems: they repeatedly rediscover corpus-level structure for each query. Modeling reusable memory construction as an MDP optimized for downstream answering performance is a compelling and potentially general framework.
  2. Strong empirical gains across multiple settings. The RL-trained builder substantially outperforms the reported retrieval, graph, autocompaction, and non-RL builder baselines on BrowseComp-Plus and also transfers to BRIGHT domains. The provenance-based fallback mechanism is particularly useful because it allows the answering agent to recover information omitted from synthetic summaries.
Critical weaknesses
  1. Insufficiently specified baseline comparisons. Although the reported gains are substantial, the comparison protocol does not provide enough detail to establish whether the baselines are comparably configured. Appendix A.7 states that source corpora, answering models, answer-generation budgets, and scoring procedures are matched, but retrieval settings, context allocation, tool access, and baseline tuning remain insufficiently documented. This makes it difficult to attribute the improvements specifically to the proposed memory-construction approach. The training-set size is also inconsistent: Section 3 and Appendix A.7 report 680 tasks, whereas Appendix A.1 reports 780.
  2. Limited evidence for practical efficiency gains. Librarian Agents incurs approximately 179K construction tokens per corpus and substantially increases answer-time token consumption relative to the base builder. The reported efficiency advantage is based on tokens per expected correct answer, rather than lower raw token consumption, and the reuse analysis primarily compares against the non-RL builder. Consequently, it does not establish when the proposed approach is more economical than strong retrieval baselines. A comparison of accuracy and total cost across query volumes, including clearly defined crossover points, is needed to substantiate the practical efficiency claims.
  3. The incremental novelty is not sufficiently established. Query-independent precomputation and amortization have been explored in Sleep-time Compute, hierarchical corpus summarization in RAPTOR, and outcome-driven RL for memory management in Memory-R1. The potentially distinctive contribution is their integration into a learned, reusable filesystem over external corpora. However, the paper lacks direct comparisons with these closely related approaches. Moreover, the RL builder produces and exposes substantially more synthetic content than the base builder, leaving unclear whether the gains arise from better organization or increased coverage and computation. Comparisons at matched memory sizes and token budgets, together with hierarchy ablations, would help establish the contribution.
Questions for authors
  1. Please clarify the BrowseComp-Plus data split. Section 3 and Appendix A.7 report 680 training tasks and 50 held-out evaluation tasks, whereas Appendix A.1 reports 780 training tasks. Which split was used for the reported results?
  2. Appendix A.7 states that source corpora, answering models, answer-generation budgets, and scoring procedures are matched. Could you additionally specify each baseline’s retrieval/context limits, tool access, tuning procedure, and construction-time and answer-time token consumption? These details would help determine whether the gains reflect improved memory construction or differences in available computation.
  3. Could you clarify the direction and interpretation of the crossover at approximately 31 questions per corpus? The paper reports an accuracy-normalized cost advantage at ten questions while raw per-query token consumption remains higher. Over which query-volume range is Librarian Agents more economical than the base builder, and how does this comparison change when using the strongest retrieval baselines?
  4. How was the reliability of the Gemini-based correctness judge assessed, particularly for BRIGHT-derived answers? Could you provide agreement with human judgments on a representative sample, or an assessment using an independent judge, to establish whether the reported gains are robust to the scoring procedure?
  5. How does the proposed approach differ substantively from Sleep-time Compute, RAPTOR, and Memory-R1 beyond their integration into a filesystem interface? Do you have comparisons or ablations at matched memory sizes and token budgets that distinguish the benefit of learned hierarchical organization from increased synthetic-content coverage?
7 comments
AuthorsOpenReview ↗

Response to Reviewer iHq6 [1/N]

We thank Reviewer iHq6 for highlighting the novelty of modeling anticipatory memory construction as an MDP alongside our strong empirical results and provenance fallback mechanism. Below, we address each of your questions with new compute-matched experiments, baseline specifications, cost crossover analyses, and cross-judge evaluations.

Construction Token Matched Comparison

Following your suggestion, we evaluated RAPTOR and an expanded-budget untuned builder on the amortization QA benchmark in Table 2:

Precomputation / Builder MethodBackbone ModelAmortization QA Accuracy
Base Builder (No-FT)Qwen 3.5-4B24.2%
Base Builder (TokenMatch)Qwen 3.5-4B24.8%
RAPTOR (TokenMatch)Qwen 3.5-4B + SBERT21.4%
Librarian Agent (Ours)Qwen 3.5-4B63.7%

Increasing the untuned builder’s construction budget improves accuracy by only 0.6 percentage points, whereas Librarian exceeds this token-matched control by 38.9 points. Librarian also exceeds token-matched RAPTOR by 42.3 points. These comparisons support the conclusion that increasing the untuned builder’s construction budget alone does not reproduce the learned builder’s gains. We distinguish this conclusion from the stronger claim that hierarchy alone explains the improvement. Matching construction tokens does not automatically match final memory size, information coverage, or answer-time consumption. The learned policy can improve which information is synthesized and how it is organized.

Contribution of organization

Appendix A.5 provides complementary content-preserving interventions. In its separate matched run, we run an ablation over original, flattened, and shuffled memories:

Memory organizationAccuracy (%)Navigation tokens
Original55.863,635
Flattened57.705,497
Shuffled51.723,612

Flattening therefore increases navigation cost by approximately 51% without lowering accuracy. Shuffling lowers point-estimate accuracy by 4.14 points at similar navigation cost. Hierarchy therefore reduces navigation cost, and semantic placement may also help accuracy, but organization alone does not explain the full gain. Note that this run uses a different question set, so its accuracies are not comparable to the 63.7% above.

Efficiency and Crossover

The reviewer correctly highlights the distinction between raw per-query token expenditure and accuracy-normalized compute efficiency. Formally, let B_m represent the fixed corpus construction cost, A_m denote the average answer-time token consumption per query, and p_m denote downstream answer accuracy. Across q queries, the cumulative token expenditure is B_m + q A_m, yielding an accuracy-normalized cost per correct answer of (B_m + q A_m) / (q p_m). Because Librarian Agents achieve significantly higher accuracy (p_m), it yields better amortized efficiency per successful outcome within an empirical crossover window of 18–54 questions (centering near 31 questions per corpus). Furthermore, increasing the construction budget of an untuned base builder fails to yield competitive accuracy, as demonstrated in our token-matched baseline experiments. Thus, the observed performance advantage stems directly from end-to-end RL policy optimization rather than redundant token allocation.

We additionally compared efficiency against Rerank and Agentic RAG on the BrowseComp+ amortization cohort. Librarian used 8.81M measured tokens in total and 36.3K tokens per correct answer, compared with 17.40M and 74.0K for Rerank and at least 9.12M and 38.8K for Agentic RAG. The total-token crossover points are 3.6 questions per corpus against Rerank and 8.4 against Agentic RAG. Thus, at the observed amortization volume, Librarian achieves accuracy statistically comparable to both strong retrieval baselines while using fewer tokens.

AuthorsOpenReview ↗

Response to Reviewer iHq6 [2/N]

Clarifying Novelty

Librarian Agents introduces anticipatory workspace construction as a learned capability: a builder transforms a corpus into a persistent, navigable filesystem before receiving downstream queries, and is trained through reinforcement learning on the answering agent’s subsequent performance. There are distinct differences from the Sleep-time Compute, RAPTOR, and Memory-R1 prior work. We will discuss each individually.

Sleep-time Compute performs unstructured chain-of-thought precomputation over short contexts. Notably, this computation isn’t a learned behavior but elicited via prompting. In contrast, in Librarian Agents we instead formulate an objective to learn how to build a persistent, navigable workspace in the form of a multi-document filesystem with explicit reference links over large corpora and formulate this as an MDP. We can view the autocompaction baseline as an instantiation of the sleep-time compute paper, which we outperform. RAPTOR relies on fixed, heuristic bottom-up embedding clustering and static summarization prompts; it is not optimized end-to-end for downstream agentic navigation, and its lossy tree compression often smooths over fine-grained cross-document relational links. We think that approaches like Raptor can be complementary to Librarian Agents as we state in Section 5, where the clustering of the builder in RAPTOR can be a good structured filesystem to further optimize with the RL fine-tuned builder from Librarian Agents. We leave this analysis to future work.

Memory-R1 studies memory management and utilization for multi-session dialogue. It incrementally takes dialogue history and maintains an external bank of extracted conversational facts using the Mem0 harness, training both a memory manager and an answering agent with outcome-driven RL, using standard policy-gradient algorithms such as PPO and GRPO. Librarian Agents instead studies upfront transformation of a document corpus into a reusable hierarchical workspace of synthetic files and source references, before downstream queries are observed. Though both methods create textual memories, a contribution here is that the agent organizes this information into an accessible data structure for future reuse. Furthermore, our training formulation not only relies on terminal answer correctness but also leverages periodic probes of improvements in answerability for better credit assignment during training. The key distinction is the scope and organization of memory construction: Memory-R1 learns incremental maintenance and retrieval-based use of conversational memories, whereas Librarian Agents learns corpus-level synthesis and organization intended to shift useful computation ahead of query time and amortize it across future queries.

To summarize, we identify three distinct components of memory building that these methods explore:

Built before queriesStructured, navigable memoryBuilder learned from downstream reward
Sleep-time ComputeYesNoNo
RAPTORYesYesNo
Memory-R1NoNoYes
Librarian AgentsYesYesYes

The three properties depend on each other. Building before queries is what allows cost to be amortized, but it means the builder cannot know what will be asked. Memory therefore has to stay navigable, so that an answering agent can find what it needs for a question nobody anticipated. And which content and organization will be useful is only revealed by downstream answers, so it has to be learned from that signal: with the same filesystem interface and the same construction budget, the untrained builder reaches 24.8% against 63.7% for Librarian, and RAPTOR's fixed structure reaches 21.4%.

Recent blogs from two industry labs, Engram [1] and Anthropic [2] (released after our paper's submission), independently isolate this capability of anticipation and "dreaming" as a core bottleneck for models at scale, finding that optimizing for hierarchical note creation improves both downstream performance and inference efficiency. This is a key problem which we begin to tackle with our Librarian Agents formulation.

AuthorsOpenReview ↗

Response to Reviewer iHq6 [3/N]

BrowseComp-Plus Data Split Clarification

All reported results use 680 training tasks, 50 validation tasks for hyperparameter/checkpoint selection, and 50 held-out evaluation tasks. The reference to “780 training tasks” incorrectly described the total dataset size as the training-set size. We will make the 680/50/50 split explicit and consistent throughout the manuscript.

Baseline configuration and reproducibility

For full reproducibility, we have released our full codebase in an anonymous repository below. Happy to share any details to make other details clear.

Anonymous URL: https://anonymous.4open.science/r/librarian-agents-anonymous-review-DB27/

Additional Legal-Domain Evaluation

We further evaluate Librarian Agents on a more realistic domain by training a Qwen 3.8-27B builder on the Harvey Legal Agent Bench [4] with the mini-swe-agent agent harness to provide our agents more flexibility with bash tools. On the benchmark, on the set of held out tasks in the validation set, training increased task-equal accuracy from 26.2% to 65.6%, placing the trained builder above all five baselines in our paper (the highest being GraphRAG at 59.15%); a frontier zero-shot Codex builder achieved 72.5%. Together with the prompt-optimization result above, this supports our claim that optimizing memory construction substantially improves a given builder, including frontier builders, rather than implying that a trained open-weight model must always outperform a much larger proprietary agent.

Model / MethodScore
Trained RL Librarian65.65%
Untrained/base model26.24%
Embedding retrieval42.01%
Reranker45.84%
GraphRAG59.15%
AutoCompaction26.98%
RAPTOR55.00%
Frontier zero-shot builder72.53%

We explicitly measure the gap between LibrarianAgents and baselines and report 95% boostrapped confidence intervals.

MethodImprovement (pp)95% CI
Embedding+23.6[11.7, 36.6]
Reranker+19.8[7.5, 34.4]
GraphRAG+6.5[1.1, 14.6]
AutoCompaction+38.7[30.3, 47.2]
RAPTOR+10.6[0.4, 21.5]

Please let us know if you find your concerns addressed, and if so, we would be grateful if you are willing to raise your score.

References

  1. Engram, Learning in a World of Notes (https://engram.com/blog/learning-in-a-world-of-notes)
  2. Anthropic, Components of Agent Memory (https://www.anthropic.com/research/components-of-agent-memory)
  3. Harvey AI, Legal Agent Bench: Law Firm Knowledge (https://www.harvey.ai/blog/legal-agent-bench-law-firm-knowledge)
Reviewer_EsT7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The framework of Librarian agents is proposed in this paper. A builder agent trained with RL to reorganize a raw document corpus into a filesystem based memory before querying. Then, a frozen answering agent navigates to answer queries. The memory is constructed and formulated via MDP whose reward is downstream answer accuracy. Experiments were run on BrowseComp-Plus and increased the strongest baseline by 33 points.

Critical strengths
  1. I think the attribution design is very clean and the ablation is well performed to back it up. Every component except the builder is frozen and training queries are hidden from the builder, and evaluation queries never touch construction or checkpoint selection. This design can help prove the paper's core claim that memory construction can be optimized end-to-end against downstream utility.
  2. It also demonstrated that transfer beyond the training distribution. The transfer results of BRIGHT tasks and comparison extended to additional domains are well supported the claim.
Critical weaknesses
  1. The 20 amortized queries per corpus are generated by Gemini 3.1 Flash Lite and graded by a Gemini 3.1 Flash Lite judge, which should separate the judge from answer agents by using different model families.
  2. I feel the confidence intervals is missing from the main accuracy result of the 50 held-out tasks. The training is reported as a single run (Fig 7) with GRPO group size 4 and such baseline can be noisy. My concern is that benchmark choice and statistical reporting may constrain the size and generality of the effect. The comparative gap between the baseline (i.e. GraphRAG, AutoCompaction) is uncertain since no per-baseline tuning details are given.
Questions for authors
  1. How were GraphRAG and AutoCompaction configured and tuned for each domain?
  2. The judge (Gemini 3.1 Flash Lite) is the same model as one of the answering agents. Could you report agreement between the judge and human grading on a manually checked subset?
  3. Please report paired bootstrap confidence intervals (or an equivalent test) for the comparisons in Fig 3 and Table 3.
  4. Was a hierarchical-summarization precomputation baseline in the RAPTOR family considered?
3 comments
AuthorsOpenReview ↗

Response to Reviewer EsT7 [1/N]

We thank Reviewer EsT7 for the positive assessment of our work and for specifically highlighting our clean experimental attribution design and out-of-distribution transfer results on BRIGHT. Below, we address each of your questions with new cross-family judge evaluations, paired bootstrap confidence intervals, baseline configuration details, and a new token-matched RAPTOR comparison.

Statistical Significance

The main gains are statistically significant under 95% paired bootstrap confidence intervals, which we computed for Figure 3 and Table 3 as you suggested: Figure 3 — Librarian versus No-FT

Answering agentPass@1 gain [95% CI]
Qwen3.5-4B+50.0 [42.0, 58.0]
Gemini 3.1 Flash Lite+49.0 [40.5, 57.0]

Thus, the improvement from RL training over the same base builder is statistically significant with both answering agents. Table 3 — Librarian versus Dense retrieval on BRIGHT

DomainQuestionsPass@1 gain [95% CI]Pass@4 gain [95% CI]
Overall150+31.2 [25.2, 37.2]+40.0 [32.7, 47.3]
AoPS50+4.5 [−5.5, 15.0]+14.0 [6.0, 24.0]
Biology50+15.0 [4.5, 25.5]+30.0 [14.0, 46.0]
Stack Overflow50+74.0 [63.0, 84.0]+76.0 [62.0, 88.0]

The pass@1 gain is significant overall and in two of three domains. On AoPS, it is significant only at pass@4.

20 amortized queries per corpus are generated and graded by the same model.

To check for self-preference or model-family bias between the query generator/answering agent (Gemini 3.1 Flash Lite) and the evaluator, we re-evaluated the amortized benchmark using a stronger, independent model family (GPT-6 Luna) as the judge.

JudgeLibrarianBudget-matched No-FTLibrarian gain [95% CI]
Gemini 3.1 Flash Lite55.9%25.7%+30.1 [21.6, 38.6]
GPT-6 Luna46.9%19.5%+27.4 [19.5, 35.0]

We see that GPT-6 Luna grades both builders more strictly, but the gain is similar (+27.4 versus +30.1 points), so the result does not depend on the judge family.

GraphRAG and AutoCompaction

Both baselines were carefully configured and tuned for each domain rather than run with naive defaults. For AutoCompaction, we followed the official implementation established in the ReasoningCache paper [1]. For GraphRAG, we followed the prompt-tuning pipeline from the Microsoft GraphRAG project [2] for entity extraction, relationship mapping, and community summarization.

All methods used the same answerer and judge. They differ only in how the corpus is prepared:

  • Retrieval: 768-token chunks with 96-token overlap, candidate depth 100, top 12 passages, at most two passages per source.
  • GraphRAG: full-corpus indexing over 6,000-character units with 300-character overlap. Each query receives up to 80 units, three community reports, and five source contexts.
  • AutoCompaction: all documents processed query-blind in 18,000-character fragments, with a rolling summary capped at 2,048 tokens. We ablated over the summary size (512, 1k, 2k, 4k, 8k) and whether the model does full re-writes or proposes diffs to the memory.

Furthermore, we release our full codebase in an anonymous repository to ensure complete reproducibility (https://anonymous.4open.science/r/librarian-agents-anonymous-review-DB27/)

AuthorsOpenReview ↗

Response to Reviewer EsT7 [2/N]

Hierarchical Summarization from RAPTOR

Following your suggestion, we implemented and evaluated RAPTOR on the amortization benchmark (Table 2). For a compute-matched comparison, we used the same backbone model as our builder (Qwen 3.5-4B) for RAPTOR's tree summarization, paired with the standard dense retriever from the original RAPTOR paper (SBERT multi-qa-mpnet-base-cos-v1). We matched the total construction token budget across RAPTOR and Librarian, and also gave the untuned base builder the same expanded budget (Base TokenMatch):

Precomputation / Builder MethodBackbone ModelAmortization QA Accuracy
Base Builder (No-FT)Qwen 3.5-4B24.2%
Base Builder (TokenMatch)Qwen 3.5-4B24.8%
RAPTOR (TokenMatch)Qwen 3.5-4B + SBERT21.4%
Librarian Agent (Ours)Qwen 3.5-4B63.7%

At the same construction budget, Librarian outperforms RAPTOR by 42.3 points. The token-matched base builder stays at 24.8%, so the gain comes from RL training rather than from the larger budget. Furthermore, we think approaches like RAPTOR can be complementary to Librarian Agents, as we state in Section 5, where RAPTOR's builder clustering can provide a structured filesystem to further optimize with the RL-fine-tuned builder from Librarian Agents. We leave such extensions and analysis to future work.

Additional Legal-Domain Evaluation

We further evaluate Librarian Agents on a more realistic domain by training a Qwen 3.8-27B builder on the Harvey Legal Agent Bench [4] with the mini-swe-agent agent harness to provide our agents more flexibility with bash tools. On the benchmark, on the set of held out tasks in the validation set, training increased task-equal accuracy from 26.2% to 65.6%, placing the trained builder above all five baselines in our paper (the highest being GraphRAG at 59.15%); a frontier zero-shot Codex builder achieved 72.5%. Together with the prompt-optimization result above, this supports our claim that optimizing memory construction substantially improves a given builder, including frontier builders, rather than implying that a trained open-weight model must always outperform a much larger proprietary agent.

Model / MethodScore
Trained RL Librarian65.65%
Untrained/base model26.24%
Embedding retrieval42.01%
Reranker45.84%
GraphRAG59.15%
AutoCompaction26.98%
RAPTOR55.00%
Frontier zero-shot builder72.53%

We explicitly measure the gap between LibrarianAgents and baselines and report 95% boostrapped confidence intervals.

MethodImprovement (pp)95% CI
Embedding+23.6[11.7, 36.6]
Reranker+19.8[7.5, 34.4]
GraphRAG+6.5[1.1, 14.6]
AutoCompaction+38.7[30.3, 47.2]
RAPTOR+10.6[0.4, 21.5]

Please let us know if you find your concerns addressed, and if so, we would be grateful if you are willing to raise your score.

References

  1. Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL (https://arxiv.org/abs/2602.03773)
  2. Graph Rag Prompt Tuning (https://microsoft.github.io/graphrag/prompt_tuning/overview/)
  3. Harvey AI, Legal Agent Bench: Law Firm Knowledge (https://www.harvey.ai/blog/legal-agent-bench-law-firm-knowledge)

I would like to thank the authors to give additional experiment results to show the statistical significance. It has addressed most of my concerns and this paper has good quality in terms of clean experimental design. It's nice to see the additional legal-domain evaluation for the librarian agents to showcase realistic application. Since this field is outside of my expertise area, I will not be able to truly assess the novelty of this work but I really like the comprehensiveness of the experimental design, therefore, recommending of acceptance.

Reviewer_VxcnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the challenge of long document context input for LLMs and solves the problem via pre-computing unstructured document corpora to efficiently serve diverse future queries. The authors propose Librarian Agents, an RL framework for building filesystem memories before future queries arrive. It trains a builder using downstream answer correctness and evaluates QA accuracy on BrowseComp-Plus and BRIGHT-derived tasks with fixed answering agents.

Critical strengths

The paper presents an RL training recipe tailored to anticipatory memory construction and demonstrates improvements in downstream QA accuracy over the evaluated baselines. Using end-to-end RL to train builder agent capability follows recent common practice of agentic RL , which provides valuable insight.

Critical weaknesses
  1. The paper provides limited attribution analysis of the RL gains. While reuse and transfer results support memory utility, further evidence is needed to justify the gains in improved anticipatory memory construction and assess possible reward hacking.

  2. The added value of specialized RL over strong general-purpose builders remains unclear. The No-FT comparison establishes a benefit for the 4B model, but does not test whether a strong zero-shot agent can construct similarly useful memories with appropriate instructions.

Questions for authors
  1. What quantitative evidence links the QA gains to improved anticipatory memory construction, and what checks address possible reward hacking?

  2. How does the RL builder compare with a strong zero-shot commercial agent, such as Claude Code with Opus, constructing reusable memory without seeing future queries?

3 comments

We thank Reviewer Vxcn for recognizing the value of our end-to-end RL recipe for anticipatory memory construction. Below, we address your questions regarding reward hacking, attribution of gains, and comparisons against strong zero-shot builders.

Attributing QA Gains & Ruling Out Reward Hacking

Our experimental design directly rules out egregious reward hacking and attributes downstream QA gains to improved anticipatory memory construction through strict out-of-distribution evaluation. Specifically, the BRIGHT evaluation domain is completely held out during RL training, featuring unseen document corpora, unseen questions, and an unseen scoring rubric. Because the builder constructs the filesystem workspace before seeing any test queries, shortcuts that exploit the training reward would not help on BRIGHT, suggesting the builder learns a transferable construction skill. Quantitatively, on the fully held-out BRIGHT evaluation, Librarian improves over Dense retrieval by 31.2 percentage points [25.2, 37.2] at pass@1 and 40.0 points [32.7, 47.3] at pass@4 overall. Furthermore, the reward is computed from a frozen, independent answering agent, and the gains hold across distinct reader models at test time, ruling out co-adaptation between the builder and a specific reader. Finally, we use BrowseComp-Plus with its unmodified evaluation harness. Any limitations in its scoring apply equally to prior methods evaluated on it [5, 6], so our comparisons to them are on equal footing.

Added Value of Specialized RL vs. Strong General-Purpose Builders

Our main claim is that end-to-end optimization substantially improves a given model's ability to construct reusable workspaces for unknown future queries, and this complements using a stronger base model. Recent publications from two industry labs, Engram [1] and Anthropic [2] released after our paper's submission, independently identify this anticipatory capability ("dreaming") as a bottleneck for frontier models, and find that optimizing hierarchical note creation improves both downstream accuracy and inference efficiency. We note that local models are often required in deployment, where users cannot send proprietary data to external APIs or need workspaces specialized to a user or domain, as highlighted by recent customizability efforts from Thinking Machines [3] and Harvey in the legal domain [4].

To answer your question directly, we added GPT-6 Luna + Codex, a frontier coding agent comparable to the one you suggest, on the amortization task (Table 2). Here, the zero-shot Codex harness achieves 79% accuracy, compared to 64% for our compact 4B RL-finetuned Librarian Agent. We cannot run RL on API-only models such as GPT-6 Luna. As a practical test of the same idea, we optimized its prompt against downstream accuracy on a validation set, which raised it from 79% to 85%.

Additional Legal-Domain Evaluation

We further evaluate Librarian Agents on a more realistic domain by training a Qwen 3.8-27B builder on the Harvey Legal Agent Bench [4] with the mini-swe-agent agent harness to provide our agents more flexibility with bash tools. On the benchmark, on the set of held out tasks in the validation set, training increased task-equal accuracy from 26.2% to 65.6%, placing the trained builder above all five baselines in our paper (the highest being GraphRAG at 59.15%); a frontier zero-shot Codex builder achieved 72.5%. Together with the prompt-optimization result above, this supports our claim that optimizing memory construction substantially improves a given builder, including frontier builders, rather than implying that a trained open-weight model must always outperform a much larger proprietary agent. |

Model / MethodScore
Trained RL Librarian65.65%
Untrained/base model26.24%
Embedding retrieval42.01%
Reranker45.84%
GraphRAG59.15%
AutoCompaction26.98%
RAPTOR55.00%
Frontier zero-shot builder72.53%

We explicitly measure the gap between LibrarianAgents and baselines and report 95% boostrapped confidence intervals.

MethodImprovement (pp)95% CI
Embedding+23.6[11.7, 36.6]
Reranker+19.8[7.5, 34.4]
GraphRAG+6.5[1.1, 14.6]
AutoCompaction+38.7[30.3, 47.2]
RAPTOR+10.6[0.4, 21.5]

Please let us know if you find your concerns addressed, and if so, we would be grateful if you are willing to raise your score.

References

  1. Engram, Learning in a World of Notes (https://engram.com/blog/learning-in-a-world-of-notes)
  2. Anthropic, Components of Agent Memory (https://www.anthropic.com/research/components-of-agent-memory)
  3. Inkling-2-Small (https://thinkingmachines.ai/news/inkling-2-small/)
  4. Harvey LAB (https://www.harvey.ai/blog/legal-agent-bench-law-firm-knowledge)
  5. Recursive Language Models (https://arxiv.org/abs/2512.24601)
  6. WebSailor: Navigating Super-human Reasoning for Web Agent (https://arxiv.org/abs/2507.02592)

Thank you for new the results and clarification. I do not have further questions now. I tend to keep the score but will also discuss with other reviewers,

AuthorsOpenReview ↗

Response to Reviewer Vxcn [1/1]

Thank you for taking the time to read our response and engage with our new results and we appreciate your time and willingness to discuss with the other reviewers.

To ensure the paper reflects these improvements, we will incorporate all of the new baselines and ablations into the revised manuscript—specifically the 27B Harvey Legal Agent Bench RL results (26.2% → 65.6%, surpassing GraphRAG and RAPTOR), the frontier zero-shot and prompt-optimized comparisons (79% → 85%), and a dedicated discussion of how OOD transfer on BRIGHT and frozen cross-reader evaluation rule out reward co-adaptation. Together, we hope these make clear that end-to-end memory optimization provides substantial gains across model scales, enabling compact open-weight builders to approach frontier agents in privacy- or cost-constrained settings where indexing large corpora via proprietary APIs is impractical.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal-Period Summary – Key Clarifications & New Results

We thank the reviewers and the AC for their thoughtful and constructive feedback. The discussion has helped us clarify the methodological contribution of Librarian Agents, strengthen the empirical comparisons, and characterize its efficiency benefits. For the AC’s convenience, we summarize the key additions and clarifications below:

  • Methodological Contribution and Dense-Reward Ablation: We clarify that our novelty comes from our task formulation of anticipatory workspace construction: learning to transform a corpus into a reusable filesystem of synthetic notes and source references, which we denote as a workspace, before observing downstream queries. We formalize the construction process as an MDP, defining a dense reward function through sequential probes that look at answerability of the partially constructed filesystem. To directly test the contribution of these periodic answerability probes, we added a terminal-reward-only RL baseline trained for the same number of steps. Accuracy increases from approximately 24% without training to 38% with terminal rewards and 64% with terminal rewards plus intermediate probes. The probes provide a 25.7-percentage-point improvement over terminal-only training (95% CI: [17.6, 34.0]), supporting their role in credit assignment during workspace construction.

  • Stronger Comparisons Controlling for Construction Budget and Memory Structure: On the amortization benchmark, Librarian achieves 63.7% accuracy, compared with 24.8% for a construction-token-matched untrained builder and 21.4% for token-matched RAPTOR. We additionally evaluated RAPTOR summaries placed into Librarian’s filesystem structure, matching the per-note size allocation and retaining source references. This stronger RAPTOR baseline reaches 30.6%, still 33.1 points below Librarian. These results show that additional construction tokens or the structural scaffold alone do not reproduce the learned builder’s gains.

  • Isolating the Contribution of Hierarchy: We clarified that the flattening ablation preserves the same synthetic content, source references, and raw documents. With generous answering limits, flattening preserves accuracy while increasing navigation tokens by approximately 51%, indicating that hierarchy primarily improves navigation efficiency in this setting. In a new comparison with a fixed 3,072-token workspace-reading budget, hierarchical memory achieves 53.33% accuracy versus 48.97% for flattened memory, a 4.37-point gain (95% CI: [0.46, 8.56]). This provides evidence that organization also benefits accuracy when reading resources are constrained.

  • Legal-Domain Evaluation and Frontier-Builder Comparisons: We add an evaluation on the Harvey Legal Agent Benchmark (LAB), where training with LibrarianAgents improves the builder from 26.24% to 65.65% task-equal accuracy, exceeding all five evaluated retrieval and memory baselines, including GraphRAG at 59.15%. We also added a frontier zero-shot Codex builder, which achieves 79% on the amortization task and 72.53% on the legal benchmark. Optimizing the frontier builder’s prompt on validation data raises its amortization accuracy to 85%, which can be viewed as a prompt-optimized variant of Librarian Agent, showcasing that there can be further improvements even on a strong builder. These comparisons sharpen our claim: optimizing workspace construction substantially improves a given builder, while stronger frontier builders can achieve higher absolute performance.

  • Statistical Reliability and Independent-Judge Evaluation: We added bootstrap confidence intervals supporting the main improvements across answering models and held-out BRIGHT domains. On BRIGHT, the overall gain over dense retrieval is 31.2 points at pass@1 (95% CI: [25.2, 37.2]). Separately, re-evaluating the amortization benchmark with an independent judge family preserves a substantial advantage over the budget-matched untrained builder: +27.4 points with GPT-6 Luna versus +30.1 points with the original Gemini judge. This supports robustness of the amortization result across judge families for both the bright and browsecomp domains;

We are confident that these additions and clarifications have addressed the major points raised during the review process. The constructive dialogue has substantially improved the paper, and we are grateful for the reviewers' engagement. We respectfully ask the AC to consider these clarifications and additional results in the final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.