acceptodds
Under review as a conference paper at ICLR 2027

Learning to Search and Searching to Learn: Joint Knowledge Internalization via Reinforcement Learning

Abstract

Integrating large language models (LLMs) with external search engines has significantly enhanced their ability to solve knowledge-intensive tasks. While search-integrated LLMs optimized via reinforcement learning (RL) excel at learning when and how to retrieve information, they typically treat retrieved text strictly as read-only context. To ensure training stability, these retrieved tokens are often masked from the policy-gradient objective, which inherently prevents models from internalizing the factual content. However, directly unmasking these tokens introduces severe noise and destabilizes the early stages of optimization. To resolve this masking dilemma, we propose LfSearch, an end-to-end RL framework that shifts the paradigm from merely learning to search toward searching to learn. LfSearch jointly optimizes search behavior and knowledge internalization through a unified dual-objective loss. By introducing an adaptive masking curriculum, the model is permitted to absorb external facts only after mastering reliable tool use. Furthermore, we apply asymmetric advantage normalization to ensure factual content is strictly reinforced rather than penalized. Across seven single-hop and multi-hop QA benchmarks, LfSearch successfully circumvents the severe performance degradation typically caused by naive unmasking. Empowered by this internalized knowledge, it consistently outperforms baselines like Search-R1, demonstrating that learned facts directly translate to superior dynamic search performance. Finally, evaluations on knowledge-retention sets explicitly confirm that LfSearch effectively distills retrieved facts into parametric memory, enabling the model to genuinely learn from its environment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Agentic Reinforcement Learning, Search-Integrated Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5nPWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LfSearch, a dual-objective framework that jointly optimizes search behavior and the internalization of retrieved knowledge into model parameters. It adaptively activates retrieved-token learning based on rollout performance and uses non-negative advantages for retrieved content. Experiments cover two models and seven QA benchmarks, with additional evaluations without retrieval to assess knowledge retention.

Critical strengths
  1. Learning from retrieved information while preserving search proficiency is a worthwhile research problem. The proposed training strategy avoids the performance degradation observed when retrieved-token masking is naively removed.

  2. The evaluation includes knowledge-retention tests without retrieval, providing complementary evidence that the model retains information encountered during training.

Critical weaknesses
  1. The improvements over the effective baseline are modest, and their reliability remains unclear. Based on the averages reported in Tables 1–2, LfSearch improves over standard Search-R1 by 1.39 and 1.63 percentage points on the two backbones. The 5.6-point improvement on Bamboogle is encouraging, but gains on several other benchmarks are small; for example, Qwen improves by only 0.04 points on TriviaQA and 0.15 points on HotpotQA. No repeated-run variance or confidence intervals are reported, making these smaller gains difficult to assess. Large improvements over the substantially degraded unmasked variant provide limited additional evidence of practical gains over standard Search-R1.

  2. The paper insufficiently situates its contribution within broader developments in search agents, and the evaluation limits its applicability. Although the related work covers several retrieval and RL methods, it does not adequately discuss complex web information-seeking approaches such as WebDancer and WebSailor. The experimental RL baselines remain limited to Search-R1 and its unmasked variant. Moreover, experiments use a fixed Wikipedia corpus with at most five interaction steps per rollout, leaving effectiveness on sustained information-seeking tasks such as BrowseComp unestablished. A controlled benchmark such as BrowseComp-Plus could assess more demanding search while retaining a fixed corpus. Although the limitations section acknowledges the restricted setting, the current evidence does not establish the method’s relevance to more complex search-agent scenarios.

Questions for authors
  1. Are the average improvements over standard Search-R1 consistent across training seeds? Could the authors provide repeated-run results or confidence intervals to clarify the reliability of the smaller gains?

  2. Does LfSearch retain its advantage on BrowseComp or BrowseComp-Plus under matched backbones, retrieval tools, and interaction budgets? In particular, how do the activation frequency and effectiveness of the knowledge-learning branch change when tasks become harder and successful trajectories become rarer?

Reviewer_wX8eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes LfSearch, a framework that jointly optimizes search capability and knowledge internalization. By attempting to internalize retrieved knowledge within a unified training framework, the proposed method outperforms Search-R1 across multiple datasets.

Critical strengths
  • Clear Presentation: The methodology and overall framework design are clearly described and easy to follow.

  • Practical Research Problem: The paper investigates how to leverage retrieved knowledge to update model parameters during search-augmented reinforcement learning, which is a practical and valuable research direction.

Critical weaknesses
  • Lack of Discussion and Comparison with Highly Related Work: The proposed approach shares significant similarities with existing literature, most notably Smart-Searcher [1]. Both works stem from the exact same motivation and utilize auxiliary losses to optimize knowledge acquisition (albeit with different loss formulations). However, the paper fails to discuss or empirically compare against this crucial baseline.

[1] Song H, Jiang J, Tian W, et al. Smart-Searcher: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning[C]//EMNLP (Findings). 2025: 13572-13586.

  • Theoretical Concerns Regarding On-Policy PPO on External Observations: The training objective is applied at the token level, including external observation tokens. However, incorporating external tokens directly into PPO training is fundamentally mismatched with the on-policy formulation. Because these tokens are sampled from an external retriever rather than the policy itself, they introduce severe off-policy bias. As highlighted in Search-R1, failing to mask out external observations inevitably leads to training instability. In contrast, Smart-Searcher intuitively mitigates this issue by rewriting the knowledge and applying a standard Negative Log-Likelihood (NLL) loss. It is unclear why the authors insist on optimizing these tokens within the on-policy RL process, and whether simply using an auxiliary NLL loss would achieve comparable results with greater stability.

  • Lack of Filtering for Knowledge Validity/Utility: The framework does not account for the actual quality or relevance of the retrieved knowledge. According to the formulation, as long as the model adheres to the output format and arrives at the correct final answer, the entire rollout is reinforced. This leaves a critical question unaddressed: how to effectively identify which specific pieces of retrieved knowledge are genuinely worth internalizing (e.g., filtering out noisy or coincidental retrievals).

  • Insufficient Empirical Evidence and Statistical Significance: The experimental results and ablation studies do not sufficiently support the paper's core claims. Specifically, compared to Search-R1, LfSearch yields only marginal average Exact Match (EM) improvements of 1.39% and 1.63% on Qwen2.5-3B and Olmo-3-7B, respectively. The gains on several individual benchmarks are minimal, with the overall average improvement being disproportionately driven by the Bamboogle dataset. Furthermore, the absence of multi-seed evaluations and standard deviation/variance reporting makes it difficult to assess the statistical significance and robustness of these improvements.

Questions for authors

see in weakness

Reviewer_D5NVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LfSearch, an end-to-end RL framework that jointly optimizes search behavior and knowledge internalization through a unified dual-objective loss. It introduces an adaptive masking curriculum and asymmetric advantage normalization to learn from retrieved content while maintaining tool-use performance. Experiments on Qwen2.5-3B and Olmo-3-7B across seven QA benchmarks show improvements over Search-R1 and naive unmasking. Evaluations on two training-derived knowledge-retention sets also show improved performance without retrieval.

Critical strengths
  1. The paper addresses a useful question: how search agents can learn from the evidence they retrieve. The adaptive masking curriculum and separate learning signals for actions and retrieved text offer a sensible way to combine tool-use learning with knowledge internalization.

  2. Results across two backbones and seven QA benchmarks show consistent gains over Search-R1. The additional no-retrieval tests on paraphrased questions and questions derived from retrieved passages provide useful evidence of knowledge retention beyond the original question wording.

Critical weaknesses
  1. A short, successful search does not mean all the retrieved text is worth memorizing. Some facts may only matter for the current question or soon become outdated, but the method does not distinguish these cases.

  2. R1-Searcher++, which also learns from retrieved knowledge during search RL, is missing from the comparisons. Without a comparison using the same backbone, training data, and retriever, it is unclear what advantage LfSearch offers over this closely related method.

  3. The performance gains may come from both memorizing retrieved knowledge and improved search behavior under the revised reward. The paper lacks ablations that isolate their respective contributions.

  4. No code is provided, making it harder to reproduce the training procedure and verify the reported results.

Questions for authors
  1. How do you determine which retrieved information is worth memorizing, especially when it is only useful once or may become outdated?
  2. How does LfSearch compare with R1-Searcher++ using the same backbone, training data, and retriever?
  3. Have you tested disabling retrieved-text learning while keeping the revised reward and other settings unchanged?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.