acceptodds
Under review as a conference paper at ICLR 2027

From Knowledge Access to Source Learning: Developing Source-Specific Competence

Abstract

Large language model (LLM) agents increasingly rely on persistent external sources to solve sequences of knowledge-intensive tasks. Existing methods improve how source content is accessed and organized, while agent-memory systems preserve reusable knowledge from prior interactions, but repeated use of the same source is still largely treated as repeated access rather than an opportunity to progressively improve understanding of that source. We study *source learning*: developing reusable *source-specific competence* over a persistent authoritative source. We represent this competence with a persistent *source model* that captures reusable understanding of the source, including how its knowledge is structured, interpreted, and applied. To construct and progressively refine such models, we propose **SourceLearn**, which combines two complementary learning mechanisms. *Self-Directed Source Learning* identifies what remains incompletely understood and adaptively revisits the source, while *Task-Guided Source Learning* uses downstream experience to reveal local representational gaps and recurring needs in how source knowledge should be organized. In both cases, learning signals determine what should be reconsidered, while persistent updates are reconstructed from the authoritative source. Across five benchmarks and three LLM backends, **SourceLearn** achieves the best performance in 13 of 15 settings, with gains of up to 22.6 points over Hybrid RAG and substantial overall improvements over static source representations and experience-based memory baselines. The code is available at [https://anonymous.4open.science/r/SourceLearn](https://anonymous.4open.science/r/SourceLearn).

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Memory, Retrieval-Augmented Generation, Source Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PCDWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SourceLearn, a framework for developing source-specific competence through a persistent, structured external source model. It combines proactive source study, task-guided refinement, and a learned representation policy that guides reconstruction from the original source. Experiments cover five benchmarks and three model backends, with SourceLearn achieving the best performance in 13 of 15 settings. Ablations and context-matched comparisons provide additional evidence of its effectiveness.

Critical strengths
  1. The evaluation covers diverse tasks, including document understanding, repository question answering, and API/tool use. Improvements across multiple backends, together with context-matched comparisons, support the framework’s practical utility.

  2. Reconstructing persistent knowledge from an authoritative source is a well-motivated design choice. Separating source knowledge from rules for organizing that knowledge also makes the framework interpretable.

Critical weaknesses
  1. The methodological novelty relative to existing experience and memory learning remains insufficiently established. ExpeL extracts reusable lessons, A-MEM organizes and evolves structured memory, and ReMe distills and refines procedural knowledge. SourceLearn’s source-grounded reconstruction and representation-policy learning are potentially distinctive, but their substantive advance over existing abstraction and memory-revision mechanisms needs clearer justification. The offline AWM comparison is useful, although a direct comparison with an evolving structured-memory approach would better substantiate this distinction.

  2. The independent benefit of learning the representation policy remains unclear. According to Appendix A.3, the variant without cross-task representation learning retains M_local, whereas the full method additionally performs recalibration under the learned policy (Eq. 14). This ablation supports the usefulness of the overall pathway, but does not distinguish the benefit of the learned policy from that of additional source rereading and reconstruction. This is a limitation in contribution attribution rather than evidence against the framework’s overall effectiveness.

Questions for authors
  1. How does learning and applying the representation policy differ operationally from storing and reusing abstracted lessons or memory-update guidance in existing approaches? A comparison with a method that revises structured memory using the same source and guidance tasks would help establish the significance of this distinction.

  2. Starting from the same M_local, how does recalibration under the learned policy Pi_1 compare with recalibration under the initial policy Pi_0, using the same source-reading and reconstruction procedure? This would help isolate the contribution of representation-policy learning and materially affect my assessment of its novelty.

4 comments
AuthorsOpenReview ↗

[Part 1/2] Response to Reviewer PCDW

We appreciate your constructive feedback and your recognition of our extensive evaluation and the motivation for source-grounded reconstruction. We provide our responses below and hope these clarifications will help you re-evaluate and update your score.

Weakness 1 & Question 1: Methodological novelty and distinction from existing memory methods.

Thank you for raising this important question. We clarify the methodological differences and provide additional comparisons with existing memory approaches.

1. Different Learning Objectives and Mechanisms.

The fundamental difference lies in what is being learned:

  • Existing memory methods primarily focus on organizing and reusing agent experiences from previous tasks. ExpeL extracts reusable insights, ReMe learns procedural knowledge from task trajectories, and A-MEM dynamically organizes and evolves memory entries.
  • SourceLearn instead treats the source itself as the object of learning, aiming to develop reusable source-specific competence rather than accumulate task experiences.

Beyond this difference in learning objectives, SourceLearn introduces a new learning framework with two complementary mechanisms: Self-Directed Source Learning proactively identifies and addresses gaps in source understanding, while Task-Guided Source Learning uses task experience to refine source representations and improve their reusability across future tasks. Both mechanisms reconstruct persistent knowledge from the authoritative source.

Thus, SourceLearn differs from existing memory approaches in both its learning objective and the mechanisms designed to achieve it.

2. How Representation-Policy Learning Differs from Abstracted Lessons.

Unlike existing approaches that retain experience-derived lessons for future tasks, SourceLearn learns a representation policy Π1\Pi_1 that guides how source knowledge is organized and represented.

Importantly, Π1\Pi_1 is applied during source-model reconstruction rather than directly provided to the task solver. It guides source-wide recalibration, including regions not covered by guidance tasks, while persistent knowledge remains grounded in the authoritative source.

3. Comparison with Existing Memory Methods.

We evaluate A-MEM, ReMe, and ExpeL using their official implementations with the same sources, guidance tasks, reference answers, and evaluation protocol (GPT-5.6-Luna, three splits):

  • A-MEM: Stores source sections and task experiences as evolving memory entries, retrieved through its agentic search.
  • ReMe + Hybrid RAG: Extracts reusable experiences from guidance-task trajectories and retrieves relevant experiences alongside Hybrid RAG evidence.
  • ExpeL + Hybrid RAG: Extracts reusable insights from guidance tasks and retrieves them alongside Hybrid RAG evidence.

A-MEM is omitted on SWE-QA because its sequential memory-evolution procedure is prohibitively expensive on large code repositories.

Accuracy (%, mean over 3 splits; two sources per benchmark):

Benchmark (sources)Hybrid RAGA-MEMReMe + Hybrid RAGExpeL + Hybrid RAGSourceLearn
MultiDoc2Dial (SSA, VA)70.257.967.572.278.2
SWE-QA (pytest, sphinx)53.4—55.969.170.1
NarrativeQA (2 sources)71.469.873.077.082.5

Insight: SourceLearn achieves the best performance across all three benchmarks, outperforming existing experience-memory and evolving structured-memory approaches. These results further support the effectiveness of developing reusable source understanding through SourceLearn's learning mechanisms, rather than relying solely on experience accumulation and memory refinement.

Revision: We have clarified the distinction from experience-based memory in the Introduction and Related Work, and the role of representation-policy learning in the Method.

AuthorsOpenReview ↗

[Part 2/2] Response to Reviewer PCDW

Weakness 2 & Question 2: Isolating the contribution of the learned representation policy Π1\Pi_1.

Thank you for this helpful suggestion. We conducted an additional Π0\Pi_0 vs. Π1\Pi_1 controlled experiment to isolate the contribution of representation-policy learning.

1. Experimental Setup.

Starting from the same MlocalM_{\mathrm{local}}, we perform one source-wide recalibration pass under two policies:

  • Generic Policy (Π0\Pi_0): The initial representation policy, without cross-task learning.
  • Learned Policy (Π1\Pi_1): The representation policy learned from guidance tasks.

Both variants use the same source, reconstruction procedure (Observe →\rightarrow Reconstruct →\rightarrow GroundApply), and computational budget. The only difference is the policy guiding reconstruction.

We evaluate six sources across three benchmarks, using three fixed splits with GPT-5.6-Luna.

2. Accuracy Results (%, mean over 3 splits).

Benchmark (sources)MlocalM_{\mathrm{local}} (no recalibration)Recalibrate(MlocalM_{\mathrm{local}}; Π0\Pi_0)Recalibrate(MlocalM_{\mathrm{local}}; Π1\Pi_1) = SourceLearn
MultiDoc2Dial (SSA, VA)75.475.878.2
SWE-QA (pytest, sphinx)65.768.170.1
NarrativeQA (2 sources)81.078.682.5

3. Insight.

The learned policy Π1\Pi_1 consistently outperforms the generic policy Π0\Pi_0 across all three benchmarks. Importantly, additional reconstruction under Π0\Pi_0 does not consistently improve over MlocalM_{\mathrm{local}}, whereas Π1\Pi_1 improves performance in every benchmark.

These results demonstrate that the learned representation policy contributes beyond additional source rereading and reconstruction, confirming the independent value of Cross-Task Representation Learning.

AuthorsOpenReview ↗

Discussion Deadline Approaching; We Welcome Further Feedback

Dear Reviewer PCDW,

We sincerely appreciate your thoughtful review and valuable feedback, which have helped us improve our manuscript. With the discussion deadline approaching, we hope that our responses and additional experiments have helped address your concerns.

If our responses have addressed your main concerns, we would be grateful if you could reconsider your rating. We also remain happy to address any additional questions or concerns.

Thank you again for your time and consideration.

Warm regards,

Authors

AuthorsOpenReview ↗

Gentle Follow-Up: Less Than 24 Hours Until the Discussion Deadline

Dear Reviewer PCDW,

With less than 24 hours remaining before the discussion deadline, we would like to kindly follow up on our previous message.

We sincerely hope that our detailed responses and additional experiments have addressed your main concerns. If so, we would greatly appreciate your reconsideration of the rating.

Thank you again for your time and consideration. We remain happy to address any further questions.

Warm regards,

Authors

Reviewer_DvuFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates an interesting problem: given a source, can the LLM gain deeper understanding of the source knowledge with repeat access? The authors propose two learning schemes: 1) self-directed source learning rereads the source and finds the deeper understanding, and 2) task-guided source learning uses downstream task to find deficiencies of current knowledge.

Critical strengths
  • In general, I like the problem this paper investigates: given a source, can the LLM gain deeper understanding of the source knowledge with repeat access.
  • The design of self-directed source learning is reasonable and easy to implement.
Critical weaknesses
  • The whole design is just fixed workflow with predefined parameters, which can limit its generalization over different models and tasks.
  • What if we simply remove the grounded reconstruction? The ablations cover learning stages and pathways, but there is no test on that removes grounded reconstruction, say writes observations and diagnoses directly into MM in the manner of a conventional memory system. This is an important comparison to show the effectness.
  • Section 5.1 states that SourceLearn "receives the same retrieved evidence as Hybrid RAG together with the activated source model," with a budget of B=24B=24k tokens. SourceLearn's task-time context is therefore a strict superset of the primary baseline's, by a large factor: Table 2 reports that matching the budget requires giving counterparts 16.5k–23.4k "evidence tokens," against Hybrid RAG's default top-8 passages.
Questions for authors
  • In Table 2, is SourceLearn's total task-time context equal to each counterpart's, or do counterparts receive 16.5k–23.4k evidence tokens while SourceLearn receives up to 24k of activated source model plus the top-8 Hybrid RAG passages?
  • Could you report the standard deviation in the Table 1 and Fig. 3?
5 comments
AuthorsOpenReview ↗

[Part 1/2] Response to Reviewer DvuF

Thank you for your thoughtful review and for recognizing the importance of our problem and the design of Self-Directed Source Learning. We address your concerns below and hope these clarifications will help you re-evaluate and update your score.

Weakness 1: Fixed workflow and predefined parameters.

Thank you for raising this concern. We would like to clarify two points:

1. General and Adaptive Learning Framework. SourceLearn is designed as a general learning framework rather than a workflow tailored to specific sources or tasks. Its two complementary mechanisms, Self-Directed Source Learning and Task-Guided Source Learning, follow a source-agnostic learning process: identifying gaps in the current source understanding, studying relevant evidence, and refining the source model accordingly. What the agent studies and how its understanding evolves are determined adaptively, rather than by predefined source-specific rules.

2. Generalization Across Sources, Tasks, and Models. The predefined parameters control the learning budget rather than what the agent learns. We use the same configuration across all 15 benchmark–backend settings, without dataset- or model-specific tuning. The evaluated sources vary substantially in type, domain, and size, providing a diverse test of the framework's generality. SourceLearn achieves the best performance in 13 of 15 settings, demonstrating strong empirical generalization across diverse sources, tasks, and model backends.

Thus, SourceLearn's learning process is general and adaptive, while its fixed computational settings do not prevent generalization across the evaluated settings.

Weakness 2: Effectiveness of Grounded Source Reconstruction.

Thank you for this suggestion. We introduce a Direct-Write ablation to evaluate the contribution of Grounded Source Reconstruction.

1. Experimental Setup. Direct-Write retains the same learning pipeline but directly stores observations and diagnosed requirements in the source model, without source rereading, grounding checks, or preservation checks during updates. All other learning components remain unchanged.

We evaluate six sources across three benchmarks, using three fixed splits with GPT-5.6-Luna.

2. Accuracy Results (%, mean over 3 splits).

Benchmark (sources)MSM_S (w/o Task-Guided)Direct-WriteSourceLearn
MultiDoc2Dial (SSA, VA)72.276.278.2
SWE-QA (pytest, sphinx)62.768.170.1
NarrativeQA (2 sources)78.681.082.5

3. Insights.

(1) Effectiveness. Direct-Write improves task accuracy over MSM_S, but SourceLearn consistently performs better across all three benchmarks, demonstrating the benefit of Grounded Source Reconstruction.

(2) Alignment with Our Learning Objective. More importantly, SourceLearn aims to improve task performance solely by developing better source understanding. Although Direct-Write can improve accuracy, it introduces task-specific information beyond the original source into the persistent model, effectively turning it into task-experience memory and violating our design principle. In contrast, Grounded Source Reconstruction uses task experience only to guide source learning, while ensuring that persistent knowledge is reconstructed from the authoritative source. We have also stated this principle explicitly in the revised Method (Section 4.4).

Weakness 3 & Question 1: Fairness of task-time context budgets.

Thank you for pointing out this ambiguity. We clarify two points:

1. Budget Clarification. The 24k-token budget is an upper limit for source-model activation, not the actual usage. Importantly, Table 2 matches the total source context (activated source model + retrieved evidence) per question, ensuring that SourceLearn receives no additional source-context tokens compared with the matched baselines.

2. Matched-Budget Evidence. While SourceLearn uses more context than the default retrieval baselines, Table 2 controls for this difference. Under matched total context budgets, retrieval baselines can retrieve more passages, while SourceLearn retains only the original top-8 evidence alongside its learned source model. SourceLearn still achieves the best accuracy on three of four sources and remains within one point of the strongest baseline on the fourth.

These results highlight the value of source understanding beyond retrieving more evidence: the learned source model provides reusable understanding of how source knowledge is structured, interpreted, and applied, complementing task-specific retrieved evidence.

Revision: We have clarified the total-context matching protocol in Appendix B.2.

AuthorsOpenReview ↗

[Part 2/2] Response to Reviewer DvuF

Question 2: Standard deviations for Table 1 and Fig. 3.

Thank you for this suggestion. We report the standard deviations for Table 1 and Fig. 3 below.

1. Table 1 with Standard Deviations.

For QA benchmarks and APIBench, we report mean ±\pm std over three fixed 30%/70% guidance/test splits.

GPT-5.6-Luna

MethodMultiDoc2DialNarrativeQASWE-QAAPIBenchAppWorld (TGC)
Hybrid RAG64.9 ±\pm 1.366.0 ±\pm 1.356.9 ±\pm 0.747.8 ±\pm 1.772.0
RAPTOR71.2 ±\pm 0.368.1 ±\pm 1.250.8 ±\pm 2.961.9 ±\pm 1.075.6
HippoRAG 266.3 ±\pm 0.366.6 ±\pm 1.962.7 ±\pm 0.755.2 ±\pm 1.577.4
AWM65.1 ±\pm 1.262.1 ±\pm 2.356.7 ±\pm 0.962.8 ±\pm 3.366.7
SourceLearn77.6 ±\pm 0.780.2 ±\pm 2.869.5 ±\pm 1.170.4 ±\pm 1.181.5

gpt-oss-120b

MethodMultiDoc2DialNarrativeQASWE-QAAPIBenchAppWorld (TGC)
Hybrid RAG64.5 ±\pm 1.045.6 ±\pm 2.152.5 ±\pm 0.460.3 ±\pm 2.630.4
RAPTOR61.5 ±\pm 1.840.4 ±\pm 0.632.8 ±\pm 1.154.7 ±\pm 0.726.8
HippoRAG 266.7 ±\pm 0.845.7 ±\pm 1.656.7 ±\pm 1.356.0 ±\pm 1.033.9
AWM64.1 ±\pm 0.343.2 ±\pm 1.851.5 ±\pm 3.163.2 ±\pm 0.328.6
SourceLearn68.7 ±\pm 0.756.7 ±\pm 3.353.4 ±\pm 0.471.8 ±\pm 1.227.4

DeepSeek-V4.1-Flash

MethodMultiDoc2DialNarrativeQASWE-QAAPIBenchAppWorld (TGC)
Hybrid RAG63.3 ±\pm 1.664.2 ±\pm 1.045.9 ±\pm 0.867.1 ±\pm 0.183.9
RAPTOR73.2 ±\pm 1.560.4 ±\pm 0.536.8 ±\pm 1.169.3 ±\pm 1.288.7
HippoRAG 267.3 ±\pm 1.362.0 ±\pm 1.055.7 ±\pm 1.369.6 ±\pm 0.586.9
AWM63.7 ±\pm 2.161.4 ±\pm 1.847.4 ±\pm 0.670.8 ±\pm 2.483.9
SourceLearn81.3 ±\pm 1.276.2 ±\pm 1.561.6 ±\pm 1.682.2 ±\pm 2.389.9

AppWorld uses the official fixed guidance/test split (90 training and 168 test-normal tasks), so the three-split protocol does not apply.

2. Fig. 3 with Standard Deviations.

We report mean ±\pm std over three fixed splits, using two sources per benchmark.

ConditionMultiDoc2DialSWE-QANarrativeQA
Full (SourceLearn, MTM_T)78.2 ±\pm 1.165.7 ±\pm 3.982.5 ±\pm 1.1
w/o Self-Directed Source Learning75.0 ±\pm 3.061.1 ±\pm 3.977.8 ±\pm 1.1
w/o Task-Guided Source Learning (MSM_S)72.2 ±\pm 2.460.3 ±\pm 3.278.6 ±\pm 1.9
Initial Source Model (M0M_0)69.0 ±\pm 2.256.6 ±\pm 3.672.6 ±\pm 3.0
w/o Failure-Guided Local Refinement (MπM_{\pi})76.2 ±\pm 1.162.5 ±\pm 4.279.8 ±\pm 1.1
w/o Cross-Task Representation Learning (MlocalM_{\mathrm{local}})75.4 ±\pm 1.562.5 ±\pm 3.981.0 ±\pm 1.9
AuthorsOpenReview ↗

Discussion Deadline Approaching; We Welcome Further Feedback

Dear Reviewer DvuF,

We sincerely appreciate your thoughtful review and valuable feedback, which have helped us improve our manuscript. With the discussion deadline approaching, we hope that our responses and additional experiments have helped address your concerns.

If our responses have addressed your main concerns, we would be grateful if you could reconsider your rating. We also remain happy to address any additional questions or concerns.

Thank you again for your time and consideration.

Warm regards,

Authors

Reviewer_2CETOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how an LLM agent can improve its performance across tasks grounded in the same persistent, authoritative knowledge source. SourceLearn maintains an explicit source model and refines it through two mechanisms: self-directed source exploration using inspection and focused study questions, and task-guided refinement using missing source requirements and aggregated representation lessons. Persistent updates are reconstructed from the original source, and the resulting model is combined with retrieved evidence at inference time. Experiments cover five benchmarks and three LLM backends, with ablations and additional analyses of representation changes, grounding, and transfer within the same source.

Critical strengths
  1. The problem setting is relevant and worth studying. Repeatedly solving tasks over the same relatively stable knowledge source is a practical setting in which independent, query-time retrieval may leave useful experience unexploited. The paper makes a useful case for studying how an agent’s persistent representation of that source should improve over time.

  2. The proposed system is coherent and systematically evaluated. The combination of source exploration, task-guided refinement, source-grounded reconstruction, and inference-time retrieval is an engineering design with a clear rationale. The evaluation spans multiple source types and model backends, and the reported improvements over the RAG baselines are substantial in many settings. The ablations and supplementary analyses help characterize the system’s behavior. The figures are also well designed and make the overall workflow easy to understand.

Critical weaknesses
  1. The methodological distinction from persistent agent memory or notebook systems is insufficiently established, making the title and framing appear stronger than the demonstrated novelty. Operationally, the system explores a source, generates focused questions, maintains structured notes, and revises those notes using task feedback and further source reading. These operations closely resemble an agent maintaining a persistent, source-grounded notebook. I appreciate the distinction that task experience identifies what to revisit while the original source supplies the persistent factual content. However, the paper does not convincingly establish whether this constitutes a substantially different mechanism or a particular design of an existing memory-based workflow. Although AWM is included, it does not directly test the closest alternative: a source-grounded notebook that supports iterative reading and feedback-driven revision. Without such a comparison, it is difficult to assess how much of the contribution lies in the proposed mechanisms versus the framing of them as “source learning.”

  2. The evaluation does not sufficiently isolate the proposed mechanisms from additional preprocessing resources and supervision. SourceLearn performs repeated source reading and adaptive exploration, and its task-guided stage uses reference answers to locate supporting evidence and derive representation requirements. These provide additional computation and supervised information relative to the retrieval and static-representation baselines. Appendix B.2 usefully controls inference-time context length, and AWM uses the same guidance tasks, so the comparisons are not uniformly mismatched. Nevertheless, these controls do not establish whether SourceLearn outperforms a simpler iterative note-refinement system given the same guidance information and a comparable preprocessing budget. The results demonstrate the effectiveness of the complete pipeline, but attribution of the gains to the specific proposed mechanisms remains incomplete.

Questions for authors
  1. Clarification of the contribution: What is the precise operational distinction between SourceLearn and an agent that maintains a persistent source-grounded notebook, rereads the source, and revises its notes based on guidance-task feedback? Which aspect of the proposed mechanism provides a substantive advance beyond that formulation?

  2. Evidence needed to assess novelty: Can the authors compare against a simple iterative notebook or summary-refinement baseline with access to the same source, guidance tasks, reference answers, and inference-time retrieval? Such a comparison would materially affect my assessment of whether the additional mechanisms are necessary.

4 comments
AuthorsOpenReview ↗

[Part 1/2] Response to Reviewer 2CET

We are grateful for your insightful review and your positive assessment of the practical relevance of our setting and the coherence of our framework. We provide our responses below and hope they will help you reconsider your evaluation and update your score.

Weakness 1 & Question 1: Methodological novelty and distinction from persistent source-grounded notebooks.

Thank you for raising this important question. We clarify SourceLearn's contributions through comparisons with existing memory methods and the proposed source-grounded notebook.

A. Compared with Existing Memory Methods: Different Learning Objectives and Mechanisms.

The fundamental difference lies in what is being learned:

  • Existing memory methods primarily focus on organizing and reusing agent experiences from previous tasks. ExpeL extracts reusable insights, ReMe learns procedural knowledge, AWM learns workflows, and A-MEM dynamically organizes and evolves memory entries.
  • SourceLearn instead treats the source itself as the object of learning, aiming to develop reusable source-specific competence rather than accumulate task experiences.

Beyond this difference in learning objectives, SourceLearn introduces a new learning framework combining Self-Directed and Task-Guided Source Learning. Both mechanisms progressively improve source understanding through grounded reconstruction from the authoritative source, rather than directly storing task-derived knowledge.

B. Beyond the Proposed Source-Grounded Notebook: Two Complementary Learning Mechanisms.

While a source-grounded notebook can reread the source and revise notes based on task feedback, SourceLearn introduces two dedicated learning mechanisms that go beyond a general read–revise workflow:

(1) Self-Directed Source Learning. Rather than simply rereading the source, SourceLearn identifies gaps in its current understanding and adaptively plans focused Deepen/Connect study actions. This enables the agent to determine what needs further study and how to study it, even before future tasks are known.

(2) Task-Guided Source Learning. Rather than directly incorporating task corrections into memory, SourceLearn uses task experience to identify what source understanding is missing and how source knowledge should be better represented. It then:

  • Locally refines source understanding by diagnosing task failures as representational gaps and reconstructing the affected source representations from the authoritative source, rather than simply incorporating task feedback.
  • Learns transferable representation principles from recurring needs across tasks and applies them to source-wide reconstruction, improving understanding even in regions not directly covered by guidance tasks.

Thus, task experience guides what and how the agent learns about the source, rather than becoming the persistent knowledge itself.

These mechanisms enable autonomous source study and transferable improvements in source understanding, beyond iterative note revision. We further evaluate their benefits through the Notebook+RAG baseline in our response to Weakness 2.

Revision: We have clarified the distinction from existing memory methods in the Introduction and Related Work, and further emphasized our two learning mechanisms in the Method.

AuthorsOpenReview ↗

[Part 2/2] Response to Reviewer 2CET

Weakness 2 & Question 2: Preprocessing budget, supervision, and matched-input notebook comparison.

Thank you for this suggestion. We introduce a Source-Grounded Notebook baseline to directly evaluate whether a simpler iterative note-refinement approach can achieve similar performance under matched resources.

1. Matched-Budget Notebook Baseline.

The notebook follows a simple iterative learning process:

  • Source Reading: Repeatedly reads source chunks and rewrites their summaries until the preprocessing budget is exhausted.
  • Task-Guided Revision: Uses guidance tasks, model responses, evaluation feedback, reference answers, and relevant source passages to revise summaries and record task experiences.
  • Further Refinement: Uses the remaining budget to revisit source chunks, prioritizing those associated with failed tasks.

For a fair comparison, both methods use the same source, guidance tasks, reference answers, preprocessing budget, and task-time context budget, with identical Hybrid RAG top-8 evidence. Budgets are matched to SourceLearn's actual usage on each source, with preprocessing costs matched separately for source reading and task-guided refinement. The notebook's actual spending stays within 1% of these budgets.

2. Accuracy Results (%).

BenchmarkHybrid RAGNotebook + RAGSourceLearn
MultiDoc2Dial (4 sources)64.972.677.6
NarrativeQA (6 sources)72.879.183.2
APIBench (TorchHub)69.669.179.9

Insight: The notebook outperforms Hybrid RAG on MultiDoc2Dial and NarrativeQA and achieves comparable performance on APIBench. This provides some evidence that progressively developing source understanding can be beneficial beyond task-time retrieval.

More importantly, SourceLearn consistently outperforms the notebook across all three benchmarks under matched supervision, learning budgets, and task-time context. These results show that SourceLearn's gains extend beyond additional resource use and simple iterative note refinement, supporting the value of its dedicated learning mechanisms.

3. Cost and Practical Value.

We acknowledge that SourceLearn incurs additional preprocessing and task-time costs. However, the initial source learning cost is incurred only once and can be amortized across future tasks. Our matched-budget experiments further show that the performance gains cannot be explained by additional computation or context alone. Given these benefits, we believe the additional costs are justified by the value of developing reusable source-specific competence.

Thanks to the author for the careful rebuttal and the additional experiment. I will update my score accordingly.

AuthorsOpenReview ↗

Thank You & Looking Forward to Further Discussion

We sincerely thank you for your positive response and willingness to update your score. We appreciate your consideration and welcome any further discussion or clarification if needed.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Reviewers,

We sincerely thank you for your thoughtful and constructive reviews. We greatly appreciate your valuable feedback, which has helped us strengthen and improve our paper. Below, we summarize the main concerns and how we have addressed them:

  1. Methodological Novelty and Distinction from Agent Memory Reviewers 2CET, PCDW We have clarified how SourceLearn differs from existing memory and notebook approaches in both its learning objective and mechanisms. In particular, Self-Directed and Task-Guided Source Learning enable adaptive source study, failure-guided refinement, and transferable improvements in source understanding, rather than simply accumulating task experiences.

  2. New Baselines and Matched-Budget Comparisons Reviewers 2CET, PCDW We have introduced a source-grounded Notebook+RAG baseline with matched supervision, preprocessing costs, and task-time context, along with additional comparisons against A-MEM, ReMe, and ExpeL. SourceLearn consistently outperforms these alternatives, supporting the benefits of its dedicated learning mechanisms beyond iterative note refinement and experience-based memory.

  3. Isolating the Contributions of Key Mechanisms Reviewers DvuF, PCDW We have conducted a Direct-Write ablation to validate Grounded Source Reconstruction and a controlled Π0\Pi_0 vs. Π1\Pi_1 experiment to isolate the contribution of Cross-Task Representation Learning. Both experiments demonstrate the value of these mechanisms beyond direct feedback incorporation or additional source reconstruction.

  4. Context Budget Fairness and Computational Costs Reviewers 2CET, DvuF We have clarified that our matched-context experiments control the total task-time source context, including both activated source-model content and retrieved evidence. We also provided detailed preprocessing budget matching for the new notebook baseline and discussed the computational costs and amortized benefits of source learning.

  5. Generalization and Statistical Reporting Reviewer DvuF We have clarified the adaptive nature of SourceLearn and its use of a unified configuration across all 15 benchmark–backend settings, without dataset- or model-specific tuning. We also reported standard deviations for Table 1 and Figure 3.

We appreciate your time and consideration, and we would be happy to address any additional questions or concerns.

Sincerely,

Authors

AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revisions

Dear Area Chair,

We sincerely appreciate your time and effort in overseeing the review process for our submission. To assist with your final assessment, we would like to briefly summarize the reviewers' feedback, our rebuttal outcomes, and the manuscript revisions.

Positive Assessments: We are grateful that the reviewers recognized several strengths of our work:

  • Problem Relevance: Reviewers 2CET and DvuF appreciated the importance and motivation of studying how agents progressively develop understanding of persistent knowledge sources.
  • Empirical Evaluation: Reviewers 2CET and PCDW highlighted our extensive evaluation across diverse sources, tasks, and model backends, as well as the substantial performance improvements.
  • Methodological Design: Reviewer 2CET acknowledged the coherence of our framework, while Reviewer PCDW recognized source-grounded reconstruction as a well-motivated design choice.

Additional Experiments and Clarifications: In response to the reviewers' concerns, we conducted several new experiments:

  • Notebook and Memory Baselines (Reviewers 2CET, PCDW): We introduced a source-grounded Notebook+RAG baseline with matched supervision, preprocessing budgets, and task-time context, along with comparisons against A-MEM, ReMe, and ExpeL. SourceLearn consistently outperformed these alternatives across the evaluated benchmarks.
  • Mechanism Validation (Reviewers DvuF, PCDW): We conducted a Direct-Write ablation to evaluate Grounded Source Reconstruction and a controlled Π0\Pi_0 vs. Π1\Pi_1 experiment to isolate the contribution of Cross-Task Representation Learning. Both experiments supported the effectiveness of these mechanisms.
  • Fairness and Generalization (Reviewers 2CET, DvuF): We clarified the matched total-context protocol, provided detailed budget comparisons, and reported standard deviations. We also highlighted SourceLearn's strong performance across 15 benchmark–backend settings using the same configuration.

Discussion Outcomes:

  • Reviewer 2CET: We are grateful that the reviewer acknowledged our careful rebuttal and additional experiments, and explicitly stated their willingness to update their score.
  • Reviewer DvuF: The reviewer acknowledged that our additional experiments addressed many questions and recognized SourceLearn's adaptive, task-agnostic nature. Their remaining concern focuses on the predefined workflow and human-inspired design. We further clarified that our framework predefines general learning strategies rather than source-specific rules or learning decisions, and explained why we believe these aspects do not inherently constitute methodological limitations.
  • Reviewer PCDW: We believe we have thoroughly addressed the reviewer's main concerns through additional memory-baseline comparisons and a controlled experiment isolating the contribution of representation-policy learning. We are awaiting further feedback from the reviewer and remain happy to address any additional questions or concerns.

Manuscript Revisions: We revised the Introduction, Related Work, and Method to clarify our contributions and distinctions from existing memory approaches, as well as Appendix B.2 to explain the context-matching protocol. Additional experiments and detailed analyses are available in our rebuttal.

We believe these clarifications and experiments have substantially strengthened our paper and addressed the major concerns raised during review. We hope this summary is helpful for your final assessment.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.