- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SourceLearn, a framework for developing source-specific competence through a persistent, structured external source model. It combines proactive source study, task-guided refinement, and a learned representation policy that guides reconstruction from the original source. Experiments cover five benchmarks and three model backends, with SourceLearn achieving the best performance in 13 of 15 settings. Ablations and context-matched comparisons provide additional evidence of its effectiveness.
Critical strengths
-
The evaluation covers diverse tasks, including document understanding, repository question answering, and API/tool use. Improvements across multiple backends, together with context-matched comparisons, support the framework’s practical utility.
-
Reconstructing persistent knowledge from an authoritative source is a well-motivated design choice. Separating source knowledge from rules for organizing that knowledge also makes the framework interpretable.
Critical weaknesses
-
The methodological novelty relative to existing experience and memory learning remains insufficiently established. ExpeL extracts reusable lessons, A-MEM organizes and evolves structured memory, and ReMe distills and refines procedural knowledge. SourceLearn’s source-grounded reconstruction and representation-policy learning are potentially distinctive, but their substantive advance over existing abstraction and memory-revision mechanisms needs clearer justification. The offline AWM comparison is useful, although a direct comparison with an evolving structured-memory approach would better substantiate this distinction.
-
The independent benefit of learning the representation policy remains unclear. According to Appendix A.3, the variant without cross-task representation learning retains M_local, whereas the full method additionally performs recalibration under the learned policy (Eq. 14). This ablation supports the usefulness of the overall pathway, but does not distinguish the benefit of the learned policy from that of additional source rereading and reconstruction. This is a limitation in contribution attribution rather than evidence against the framework’s overall effectiveness.
Questions for authors
-
How does learning and applying the representation policy differ operationally from storing and reusing abstracted lessons or memory-update guidance in existing approaches? A comparison with a method that revises structured memory using the same source and guidance tasks would help establish the significance of this distinction.
-
Starting from the same M_local, how does recalibration under the learned policy Pi_1 compare with recalibration under the initial policy Pi_0, using the same source-reading and reconstruction procedure? This would help isolate the contribution of representation-policy learning and materially affect my assessment of its novelty.
4 comments
[Part 2/2] Response to Reviewer PCDW
Weakness 2 & Question 2: Isolating the contribution of the learned representation policy .
Thank you for this helpful suggestion. We conducted an additional vs. controlled experiment to isolate the contribution of representation-policy learning.
1. Experimental Setup.
Starting from the same , we perform one source-wide recalibration pass under two policies:
- Generic Policy (): The initial representation policy, without cross-task learning.
- Learned Policy (): The representation policy learned from guidance tasks.
Both variants use the same source, reconstruction procedure (Observe Reconstruct GroundApply), and computational budget. The only difference is the policy guiding reconstruction.
We evaluate six sources across three benchmarks, using three fixed splits with GPT-5.6-Luna.
2. Accuracy Results (%, mean over 3 splits).
| Benchmark (sources) | (no recalibration) | Recalibrate(; ) | Recalibrate(; ) = SourceLearn |
|---|---|---|---|
| MultiDoc2Dial (SSA, VA) | 75.4 | 75.8 | 78.2 |
| SWE-QA (pytest, sphinx) | 65.7 | 68.1 | 70.1 |
| NarrativeQA (2 sources) | 81.0 | 78.6 | 82.5 |
3. Insight.
The learned policy consistently outperforms the generic policy across all three benchmarks. Importantly, additional reconstruction under does not consistently improve over , whereas improves performance in every benchmark.
These results demonstrate that the learned representation policy contributes beyond additional source rereading and reconstruction, confirming the independent value of Cross-Task Representation Learning.
Discussion Deadline Approaching; We Welcome Further Feedback
Dear Reviewer PCDW,
We sincerely appreciate your thoughtful review and valuable feedback, which have helped us improve our manuscript. With the discussion deadline approaching, we hope that our responses and additional experiments have helped address your concerns.
If our responses have addressed your main concerns, we would be grateful if you could reconsider your rating. We also remain happy to address any additional questions or concerns.
Thank you again for your time and consideration.
Warm regards,
Authors
Gentle Follow-Up: Less Than 24 Hours Until the Discussion Deadline
Dear Reviewer PCDW,
With less than 24 hours remaining before the discussion deadline, we would like to kindly follow up on our previous message.
We sincerely hope that our detailed responses and additional experiments have addressed your main concerns. If so, we would greatly appreciate your reconsideration of the rating.
Thank you again for your time and consideration. We remain happy to address any further questions.
Warm regards,
Authors
[Part 1/2] Response to Reviewer PCDW
We appreciate your constructive feedback and your recognition of our extensive evaluation and the motivation for source-grounded reconstruction. We provide our responses below and hope these clarifications will help you re-evaluate and update your score.
Thank you for raising this important question. We clarify the methodological differences and provide additional comparisons with existing memory approaches.
1. Different Learning Objectives and Mechanisms.
The fundamental difference lies in what is being learned:
Beyond this difference in learning objectives, SourceLearn introduces a new learning framework with two complementary mechanisms: Self-Directed Source Learning proactively identifies and addresses gaps in source understanding, while Task-Guided Source Learning uses task experience to refine source representations and improve their reusability across future tasks. Both mechanisms reconstruct persistent knowledge from the authoritative source.
Thus, SourceLearn differs from existing memory approaches in both its learning objective and the mechanisms designed to achieve it.
2. How Representation-Policy Learning Differs from Abstracted Lessons.
Unlike existing approaches that retain experience-derived lessons for future tasks, SourceLearn learns a representation policy Π1 that guides how source knowledge is organized and represented.
Importantly, Π1 is applied during source-model reconstruction rather than directly provided to the task solver. It guides source-wide recalibration, including regions not covered by guidance tasks, while persistent knowledge remains grounded in the authoritative source.
3. Comparison with Existing Memory Methods.
We evaluate A-MEM, ReMe, and ExpeL using their official implementations with the same sources, guidance tasks, reference answers, and evaluation protocol (GPT-5.6-Luna, three splits):
A-MEM is omitted on SWE-QA because its sequential memory-evolution procedure is prohibitively expensive on large code repositories.
Accuracy (%, mean over 3 splits; two sources per benchmark):
Insight: SourceLearn achieves the best performance across all three benchmarks, outperforming existing experience-memory and evolving structured-memory approaches. These results further support the effectiveness of developing reusable source understanding through SourceLearn's learning mechanisms, rather than relying solely on experience accumulation and memory refinement.
Revision: We have clarified the distinction from experience-based memory in the Introduction and Related Work, and the role of representation-policy learning in the Method.