acceptodds
Under review as a conference paper at ICLR 2027

DeepRefine: Agentic Knowledge Refinement via Reinforcement Learning

Abstract

External knowledge enables large language model (LLM) agents to ground their actions and decisions beyond intrinsic parametric memory in open-ended, knowledge-intensive downstream tasks. Yet the quality of the underlying knowledge bases is systematically limited by incompleteness, incorrectness, or redundancy, manifested as missing evidence or cross-document links, low-confidence or imprecise claims, and ambiguous or coreference resolution issues. Such defects compound under iterative use, degrading retrieval fidelity and downstream task performance. We present DeepRefine, a reinforcement learning framework for agentic knowledge refinement that evolves the quality of any pre-constructed structured knowledge bases, e.g., knowledge graphs or LLM-Wikis, with user queries to make it more suitable for the downstream tasks. DeepRefine performs multi-turn interactions with the knowledge base and conducts abductive diagnosis over the interaction history, localizes likely defects, and executes targeted refinement actions for incremental knowledge base updates. To further optimize refinement policies of DeepRefine without golden refinement trajectories, we introduce a Gain-Beyond-Draft (GBD) reward and train the reasoning process end-to-end via reinforcement learning. Extensive experiments demonstrate consistent downstream gains over strong baselines.

Primary area:
reinforcement learning
Keywords:
large language models, knowledge base, agentic reinforcement learning, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ycWhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose to train, using an RL framework, a policy that refines a knowledge base in a query dependent manner. In order to do so they construct, using query answer pairs, a reward based on the tokenwise F1 of the ground truth when using the refined knowledge base to answer the query via RAG. In order to produce the candidate refinements they design a judgement-abduction-refinement harness/scaffolding/prompt.

Critical strengths

The combination of (1) agentic KB refinement (i.e. with an existing KB) and (2) RL using query answer pairs is, to the best of my knowledge novel. It is also timely since despite the extensive work in agentic KB generation, (imperfect) KBs are readily available in many settings. Both challenges that the paper puts forward in the introduction (lack of GT refinement trajectories via RL, and the guided exploration of the KG via their heuristic design) are well motivated and addressed by their method.

Critical weaknesses

I find that both agentic KG refinement (e.g.[1, 2] ) and RL methods for KG refinement/editing (e.g. [3]) are missing in the related work discussion. This is not a novelty weakness since I do not know of any work that has all of the components of the method proposed in the current manuscript and its same settings. However, I believe that these (and other) existing works should be better contrasted in the manuscript with the proposed approach to better underline its novelty and scope. In the same vein, I believe that including a learned KB adaptation method in the main experimental results would strengthen the experimental claims.

[1] LLMs as Knowledge Graph Refiners: Mitigating Factual Inconsistencies in Generative Knowledge Extraction. Kim et al., ACL 2026

[2] TRAIL: Joint Inference and Refinement of Knowledge Graphs with Large Language Models. Zhao et. al arXiv preprint 2025.

[3] KGER: Knowledge Graph Error Detection and Refinement with Reinforcement Learning A. Maoliniyazi ICASSP 2026

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer ycWh (1/1)

Thank you for recognizing the value and novelty of our agentic KB refinement method with RL. Also thanks for your recognition about our supported claim!

We appreciate the references and have contrasted these methods more clearly in the revised version.

GraphRefine [1]  fine-tunes an LLM to correct triple-level inconsistencies between a generated KG and its source text. DeepRefine instead uses queries to locate and edit potentially problematic KB regions, and learns its refinement policy from downstream QA feedback without requiring gold edit trajectories.

TRAIL [2] uses confidence-based validation to incorporate and refine facts generated during reasoning. DeepRefine instead learns to repair an existing KB using downstream feedback, without gold edit trajectories. The distinction is therefore fact-confidence-driven knowledge integration versus task-utility-driven KB refinement.

KGER [3]  uses RL for error detection and correction over graph-specific entity and relation embeddings. Its released implementation accepts a triple or replaces its head, relation, or tail using a predefined entity/relation vocabulary. DeepRefine instead targets query-conditioned refinement of open-text structured KBs using downstream QA feedback. Applying KGER to this setting would require adapting its representation and correction pipeline.

We did not find official public implementations for GraphRefine or TRAIL. And KGER’s implementation is public, but its method can not be generalized to our open-text structured KBs. Our current results therefore do not constitute a direct comparison with these methods. We have discussed them in the related work of our revision.

[1] LLMs as Knowledge Graph Refiners: Mitigating Factual Inconsistencies in Generative Knowledge Extraction. Kim et al., ACL 2026

[2] TRAIL: Joint Inference and Refinement of Knowledge Graphs with Large Language Models. Zhao et. al arXiv preprint 2025.

[3] KGER: Knowledge Graph Error Detection and Refinement with Reinforcement Learning A. Maoliniyazi ICASSP 2026

Dear Reviewer ycWh,

As the discussion deadline is approaching, we wanted to kindly ask whether our response has addressed your concerns. We are happy to provide any further clarification before the discussion closes.

Thank you again for your time.

Reviewer_AVPXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DeepRefine, a reinforcement learning framework for query-driven refinement of pre-constructed knowledge bases. The method combines multi-turn answerability judgments, error abduction, and targeted editing operations, including inserting edges, deleting edges, and replacing nodes. The refinement policy is trained using GRPO with a Gain-Beyond-Draft (GBD) reward measuring the improvement in downstream answer F1 after refinement. Experiments cover five datasets, multiple knowledge-base constructors, and several retrieval methods. The results demonstrate useful downstream improvements, although the research gap are not sufficiently established.

Critical strengths
  1. The paper addresses a practically relevant problem. Improving an existing knowledge base without reconstructing it is valuable, particularly when knowledge construction is expensive and user interactions reveal previously unnoticed deficiencies.

  2. The framework is reasonable and clearly structured. Separating evidence gathering, error analysis, and refinement provides an understandable approach to knowledge-base maintenance. The ablations support the usefulness of RL training and the error-abduction step.

Critical weaknesses
  1. The research gap is not sufficiently distinguished from existing work. The introduction explains why imperfect knowledge bases need refinement and why full reconstruction can be expensive. However, these practical motivations do not clearly establish what existing knowledge-refinement, agent-memory, or iterative graph-editing methods cannot accomplish. Section 2.3 acknowledges methods such as TRAIL that already support iterative KG updates. It remains unclear whether the central contribution is better defect localization, downstream-reward optimization of editing, or persistent improvements that generalize across queries. These are different research claims and require more explicit positioning and corresponding comparisons.
  2. The comparisons do not fully isolate the value of persistent refinement from additional inference computation. DeepRefine introduces extra retrieval, reasoning, and editing calls. A matched-budget baseline that performs additional retrieval and reasoning without modifying the shared KB would help establish when persistent editing is preferable.
Questions for authors
  1. What specific limitation of the closest iterative KG-editing and agent-memory methods does DeepRefine overcome, beyond combining query-conditioned editing with standard RL training?

  2. Under Algorithm 2, how does GBD differ from directly using refined-answer F1 as the reward? Are there implementation details that prevent the shared draft baseline from cancelling during group normalization?

  3. Can the authors evaluate a strictly separated refinement-query set and future-query set, or a chronological protocol in which each query is answered before its resulting edits affect subsequent queries?

  4. How does DeepRefine compare with a matched-budget retrieval-and-reasoning baseline that does not persist edits to the KB?

  5. What fraction of edits are factually correct, and can repeated refinement harm previously answerable questions or knowledge unrelated to the refinement-driving queries?

8 comments
AuthorsOpenReview ↗

Response to Reviewer AVPX (1/4)

We sincerely thank Reviewer AVPX for the valuable and constructive comments, as well as for pointing out the practically relevant problem addressed in our paper, and the reasonable and clearly structured framework of our framework.

We would like to address your concerns individually below:

The research gap is not sufficiently distinguished from existing work.

We thank the reviewer for the opportunity to clarify the research gap. DeepRefine targets a specific and practically important setting:  how to improve an already constructed, shared structured KB through reusable local edits, using downstream task feedback rather than gold editing trajectories or full KB reconstruction. Our contribution is a framework for this setting, which is not the individual observation that KG can be updated or that RL can be used for memory management.

Positioning relative to iterative KG editing. TRAIL[1] demonstrates that reasoning and KG updates can be interleaved, using confidence-driven mechanisms to generate, validate, and prune facts. DeepRefine addresses a complementary question:  how can a refinement policy learn which changes to make to an existing KB from downstream interaction histories?  To this end, it combines query-conditioned interaction, defect abduction, and refinement actions generation, with the edits applied to the shared KB. This targets the utility of the refined KB for downstream use, rather than treating graph updating only as a step within the current reasoning process.

Positioning relative to agent memory. RL-based memory methods such as Memory-R1[2] already learn memory operations on newly introduced memory from downstream feedback. DeepRefine focuses on a different, though related, object: a pre-constructed structured KB whose existing entities, relations, and connections can be locally refined. Its action space and reasoning process are designed around locating and editing potentially defective graph neighborhoods, rather than only deciding how to manage individual newly introduced memory entries. We will clarify this distinction in our revision.

Value beyond the refinement queries.  A key objective is that edits produced from historical queries remain useful for other queries. To test this more directly on multi-hop benchmarks, we additionally split the queries into disjoint historical and future sets. The selector and refinement pipeline access only historical queries and the full, fixed initial KB. Future queries are used only for evaluation.

Methods2WikiQAMusiqueAverage
Subgraph (2-hop)48.7221.2034.96
Subgraph (2-hop) + DeepRefine-4B49.94$_{+1.22}$23.85$_{+2.65}$36.90$_{+1.94}$
Subgraph (2-hop) + DeepRefine-8B50.46$_{+1.74}$24.51$_{+3.31}$37.49$_{+2.53}$
ToG46.6718.1632.42
ToG + DeepRefine-4B46.23$_{-0.44}$23.34$_{+5.18}$34.79$_{+2.37}$
ToG + DeepRefine-8B46.92$_{+0.25}$23.69$_{+5.53}$35.31$_{+2.89}$

These results provide direct evidence that, in the tested settings, historical-query-driven refinements improve performance on queries unavailable during selection and refinement, across both evaluated retrievers. Overall, DeepRefine studies interactions-guided, post-construction refinement of a shared structured KB. Defect abduction is a mechanism, and cross-query reuse is an empirical evidence.

[1] TRAIL: Joint Inference and Refinement of Knowledge Graphs with Large Language Models. Zhao et. al arXiv preprint 2025.

[2] Yan S, Yang X, Huang Z, et al. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 12805-12825.

AuthorsOpenReview ↗

Response to Reviewer AVPX (2/4)

Follows up to Response to Reviewer AVPX (1/4).

how does GBD differ from directly using refined-answer F1 as the reward?

Thank you for pointing out that. We agree that if the draft F1 scores within a group are the same, GBD rewards would not introduce differences from refined-answer F1 alone. However, we would like to clarify that in our implementations, the draft F1 scores are recomputed per rollout and could be different in a same group. We apologize for the confusion in Algorithm 2 and initially released code, and we have fixed that in our revision.

To be specific, different global random seeds are used for RAG-based reward calculating across rollouts in a group to expose training to multiple realizations of RAG randomness, rather than relying on a single fixed global seed. Within each rollout, the draft and refined evaluations shared a global seed to facilitate a paired comparison. This design was motivated by the goal of making refinement-gain measurement less dependent on a particular RAG realization. Beyond its effect on optimization, compared with absolute scores, GBD provides an explicit and interpretable measure of the observed performance gain or degradation relative to the draft KB. We will clarify this distinction in the revision.

To further demonstrate that varying the global random seeds of the RAG processes used for reward computation across different rollouts within the same group can lead to different F1 scores, we evaluate 50 examples selected from the training data. For each example, we keep the draft KB and all model and retrieval configurations fixed, using Qwen3-Embedding-0.6B\texttt{Qwen3-Embedding-0.6B} as the retriever, Qwen2.5-7B-Instruct\texttt{Qwen2.5-7B-Instruct} as the reader, top-10 retrieval, and a decoding temperature of 0. For robustness, we repeat the evaluation using three independently sampled, non-overlapping random-seed groups, each containing 12 global seeds. As shown in the following table, between 9 and 11 out of 50 examples produce different F1 scores across seeds, corresponding to variation rates of 18%-22% and an average of 19.3%.

Seed GroupSeeds per ExampleSame F1Different F1Difference Rate
Group 112391122%
Group 21241918%
Group 31241918%
Average1240.39.719.3%

We also trained a model DeepRefine-4B w/ F1 Reward and evaluated it on 5 benchmarks under the naive constructor and two triple retrievers to demonstrate the gains brought from GBD reward design.

VariantsRetrieversNQPopQA2WikiQAMusiqueLOCOMOOverall
-Subgraph26.4354.4833.5313.8225.8630.82
DeepRefine-4B w/ GBD RewardSubgraph29.5560.0936.0415.9329.6434.25$_{+3.43}$
DeepRefine-4B w/ F1 RewardSubgraph28.1057.7535.3613.6527.9032.55$_{+1.73}$
-ToG25.5554.9243.7418.2122.6433.01
DeepRefine-4B w/ GBD RewardToG29.2159.2749.1625.7324.5337.58+4.57_{+4.57}
DeepRefine-4B w/ F1 RewardToG28.3754.8940.3120.0520.1432.75$_{-0.26}$

As the table shows, the performance of DeepRefine trained with F1 score as reward is obviously lower than DeepRefine trained with GBD reward. And interestingly, DeepRefine w/ F1 reward perform very bad on ToG retriever. It may because our RAG evaluation in reward calculations uses the Subgraph retriever and with F1 score as the reward, the policy only learns to optimize its performance under the randomness of Subgraph retriever. However, with GBD reward, the rewards are exactly from the gains, will not be influenced very much by the randomness of the RAG process during the reward calculations, which is a more stable design than F1 reward. We have added that results in Appendix F of our revision.

AuthorsOpenReview ↗

Response to Reviewer AVPX (3/4)

Follows up to Response to Reviewer AVPX (2/4).

Evaluate a strictly separated refinement-query set and future-query set, or a chronological protocol in which each query is answered before its resulting edits affect subsequent queries?

We thank the reviewer for this observation. We would like to first clarify that the held-out queries in Appendix E are not used to drive refinement actions. Although we conduct coverage-based query selections before splitting the held-out query, we want to make sure the queries in the refinement set can be associated with a relatively large coverage of the full KB content, which makes the held-out queries can benefit more to the refinement process.

Furthermore, we still conduct an additional historical-to-future query evaluation. We first split the benchmark queries into two disjoint sets: a historical query set with 500 queries and a future query set with another 500 queries. The coverage selector has access only to the historical queries, while retrieving candidate triples from the full, fixed initial KB. We then refine the shared KB using only the selected historical queries. The future queries are used solely for evaluation and are not available during selection or refinement. We evaluate the same future queries on both the draft and refined KBs using the same retriever, reader, and scoring procedure as those in our paper.

Methods2WikiQAMusiqueAverage
Subgraph (2-hop)48.7221.2034.96
Subgraph (2-hop) + DeepRefine-4B49.94$_{+1.22}$23.85$_{+2.65}$36.90$_{+1.94}$
Subgraph (2-hop) + DeepRefine-8B50.46$_{+1.74}$24.51$_{+3.31}$37.49$_{+2.53}$
ToG46.6718.1632.42
ToG + DeepRefine-4B46.23$_{-0.44}$23.34$_{+5.18}$34.79$_{+2.37}$
ToG + DeepRefine-8B46.92$_{+0.25}$23.69$_{+5.53}$35.31$_{+2.89}$

As shown in the above table, under the rigorous online KB evolution settings, the refinement actions generated from DeepRefine can still benefit future queries in most cases. We have added these experimental into Appendix E of our revision.

How does DeepRefine compare with a matched-budget retrieval-and-reasoning baseline that does not persist edits to the KB?

Thank you for this question. DeepRefine is designed to be complementary to retrieval-and-reasoning methods, rather than a replacement for them. Retrieval and reasoning improve how a system uses the current KB to answer a query, whereas DeepRefine improves the shared KB itself, allowing the resulting edits to be reused by subsequent queries.

Importantly, DeepRefine does not require the kind of extended reasoning trajectory produced by methods such as ReAct. Its multi-turn interaction is primarily based on simple retrieval-level subgraph expansion, with an answerability judgment at each interaction. This history is then used for defect abduction and refinement action generation. Our results show that such a relatively simple interaction history is already sufficient to produce KB edits that benefit downstream QA. Richer histories from retrieval-and-reasoning systems could potentially provide additional evidence for refinement, making them a promising complementary source of information.

Reviewer_4ocpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DeepRefine improves an existing structured knowledge base through query-guided edits. It checks answerability, diagnoses possible defects, and generates edge insertions, edge deletions, or node replacements. The policy is trained with GRPO using the change in downstream answer F1 as its reward. Experiments cover two policy sizes, three constructors, four retrievers, and five QA datasets, with additional ablations and held-out-query evaluation.

Critical strengths
  1. Post-construction refinement addresses a useful problem. Local edits offer a practical way to improve an existing KB when rebuilding it is expensive. The separation of answerability assessment, diagnosis, and editing makes the approach easy to follow.

  2. The evaluation covers varied construction and retrieval settings. Table 1 reports average F1 gains for every constructor–retriever combination. The larger gains for triple retrievers are consistent with the method's focus on graph edits. Table 2 also shows lower refinement costs than full reconstruction in the evaluated setting.

  3. The held-out evaluation provides useful evidence of transfer. Appendix E reports gains on queries excluded from the refinement subset. This supports the value of the resulting KB edits beyond the queries that directly prompted them.

Critical weaknesses
  1. GBD does not change the advantages in the stated GRPO update. Algorithm 2 subtracts the same draft score from every rollout for a query. For rewards Ri=fi−bqR_i=f_i-b_q, group normalization gives (Ri−Rˉ)/std⁡(R)=(fi−fˉ)/std⁡(f)(R_i-\bar R)/\operatorname{std}(R)=(f_i-\bar f)/\operatorname{std}(f). Under this algorithm, GBD therefore produces the same advantages as refined-answer F1 alone. Its contribution as a distinct training objective is unclear.

  2. The policy's access to source text is unclear. Sections 4.1 and 4.3 describe refinement using the query and retrieved KB evidence, but Figure 12 also supplies original_text. The paper does not clearly specify how this text is selected at evaluation time. This matters because source passages may provide facts absent from the retrieved graph.

  3. The evaluation does not directly test online KB evolution. The coverage selector uses the full benchmark query pool before refinement. Appendix E usefully excludes refinement queries from scoring, but still uses this selector. These results support reuse across queries, while leaving the setting in Section 4.3—where future queries are unavailable—less well established.

  4. Improved answer F1 does not directly establish edit correctness. The results show that the edited KB is more useful for the evaluated QA tasks, but provide limited systematic evidence that individual edits are supported by the source documents. Since these edits persist, their correctness is an important part of the claimed improvement in knowledge quality.

Questions for authors
  1. Does the implementation contain any step that prevents the draft baseline from cancelling during GRPO normalization? If not, could you clarify GBD's role relative to using refined-answer F1 directly?

  2. How is original_text in Figure 12 selected during evaluation, and what source passages can the policy access beyond the retrieved triples?

  3. Would the held-out gains persist if the refinement pool were fixed before the evaluation queries became available? A split made before coverage selection would directly test this claim.

  4. Could you audit a small sample of edits against the source documents and report how often the changes are justified?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4ocp (1/3)

We sincerely thank Reviewer 4ocp for the valuable and constructive comments, as well as for pointing out the useful problem addressed in our paper, evaluations under varied settings, and useful evidences of transfer.

We would like to address your concerns individually below:

W1 & Q1: GBD does not change the advantages in the stated GRPO update

Thank you for pointing out that. We agree that if the draft F1 scores within a group are the same, GBD rewards would not introduce differences from refined-answer F1 alone. However, we would like to clarify that in our implementations, the draft F1 scores are recomputed per rollout and could be different in a same group. We apologize for the confusion in Algorithm 2 and initially released code, and we have fixed that in our revision.

To be specific, different global random seeds are used for RAG-based reward calculating across rollouts in a group to expose training to multiple realizations of RAG randomness, rather than relying on a single fixed global seed. Within each rollout, the draft and refined evaluations shared a global seed to facilitate a paired comparison. This design was motivated by the goal of making refinement-gain measurement less dependent on a particular RAG realization. Beyond its effect on optimization, compared with absolute scores, GBD provides an explicit and interpretable measure of the observed performance gain or degradation relative to the draft KB. We will clarify this distinction in the revision.

To further demonstrate that varying the global random seeds of the RAG processes used for reward computation across different rollouts within the same group can lead to different F1 scores, we evaluate 50 examples selected from the training data. For each example, we keep the draft KB and all model and retrieval configurations fixed, using Qwen3-Embedding-0.6B\texttt{Qwen3-Embedding-0.6B} as the retriever, Qwen2.5-7B-Instruct\texttt{Qwen2.5-7B-Instruct} as the reader, top-10 retrieval, and a decoding temperature of 0. For robustness, we repeat the evaluation using three independently sampled, non-overlapping random-seed groups, each containing 12 global seeds. As shown in the following table, between 9 and 11 out of 50 examples produce different F1 scores across seeds, corresponding to variation rates of 18%-22% and an average of 19.3%.

Seed GroupSeeds per ExampleSame F1Different F1Difference Rate
Group 112391122%
Group 21241918%
Group 31241918%
Average1240.39.719.3%

We also trained a model DeepRefine-4B w/ F1 Reward and evaluated it on 5 benchmarks under the naive constructor and two triple retrievers to demonstrate the gains brought from GBD reward design.

VariantsRetrieversNQPopQA2WikiQAMusiqueLOCOMOOverall
-Subgraph26.4354.4833.5313.8225.8630.82
DeepRefine-4B w/ GBD RewardSubgraph29.5560.0936.0415.9329.6434.25$_{+3.43}$
DeepRefine-4B w/ F1 RewardSubgraph28.1057.7535.3613.6527.9032.55$_{+1.73}$
-ToG25.5554.9243.7418.2122.6433.01
DeepRefine-4B w/ GBD RewardToG29.2159.2749.1625.7324.5337.58+4.57_{+4.57}
DeepRefine-4B w/ F1 RewardToG28.3754.8940.3120.0520.1432.75$_{-0.26}$

As the table shows, the performance of DeepRefine trained with F1 score as reward is obviously lower than DeepRefine trained with GBD reward. And interestingly, DeepRefine w/ F1 reward perform very bad on ToG retriever. It may because our RAG evaluation in reward calculations uses the Subgraph retriever and with F1 score as the reward, the policy only learns to optimize its performance under the randomness of Subgraph retriever. However, with GBD reward, the rewards are exactly from the gains, will not be influenced very much by the randomness of the RAG process during the reward calculations, which is a more stable design than F1 reward. We have also add that comparisons in Appendix F.

AuthorsOpenReview ↗

Response to Reviewer 4ocp (2/3)

Follows up to Response to Reviewer 4ocp (1/3).

W2 & Q2: The policy's access to source text is unclear.

Thank you for your feedbacks! We would like to clarify that during the structured KB construction with whatever base KG constructor, RL fine-tuned KG constructor, or Graphify constructor, each entity are associated with one or more its source text chunks. It is a common construction methods that can be found in many past works [1][2]. During the refinement process, the original_text is obtained from the source text chunk set, where the text chunks are associated with the entities in the retrieved triples during refinement. This process is also similar to the way to obtain retrieved passages in plenty of GraphRAG methods (e.g., HippoRAG[1]) with passage retrievers. Besides, to avoid there are too many associated source chunks in the policy's context, which could lead to efficiency and long context truncation issues, we select top-15 relevant source text chunks to add into the context, ranked by the similarity scores of their associated entities. This design aims to let the policy model know the source information and can correct any counterfactual issues and can also refine with references.

Thank you again for pointing out this. We have added that descriptions in the Appendix J of our revision.

[1] Gutiérrez B J, Shu Y, Gu Y, et al. Hipporag: Neurobiologically inspired long-term memory for large language models[J]. Advances in neural information processing systems, 2024, 37: 59532-59569.

[2] Edge D, Trinh H, Cheng N, et al. From local to global: A graph rag approach to query-focused summarization[J]. arXiv preprint arXiv:2404.16130, 2024.

W3 & Q3: The evaluation does not directly test online KB evolution.

We thank the reviewer for this observation. We would like to first clarify that we conduct coverage-based query selections before splitting the held-out query because we want to make sure the queries in the refinement set can be associated with a relatively large coverage of the full KB content, which makes the held-out queries can benefit more to the refinement process. The held-out queries in Appendix E are not used to drive refinement actions.

Furthermore, we still conduct an additional historical-to-future query evaluation. We first split the benchmark queries into two disjoint sets: a historical query set with 500 queries and a future query set with another 500 queries. The coverage selector has access only to the historical queries, while retrieving candidate triples from the full, fixed initial KB. We then refine the shared KB using only the selected historical queries. The future queries are used solely for evaluation and are not available during selection or refinement. We evaluate the same future queries on both the draft and refined KBs using the same retriever, reader, and scoring procedure as those in our paper.

Methods2WikiQAMusiqueAverage
Subgraph (2-hop)48.7221.2034.96
Subgraph (2-hop) + DeepRefine-4B49.94$_{+1.22}$23.85$_{+2.65}$36.90$_{+1.94}$
Subgraph (2-hop) + DeepRefine-8B50.46$_{+1.74}$24.51$_{+3.31}$37.49$_{+2.53}$
ToG46.6718.1632.42
ToG + DeepRefine-4B46.23$_{-0.44}$23.34$_{+5.18}$34.79$_{+2.37}$
ToG + DeepRefine-8B46.92$_{+0.25}$23.69$_{+5.53}$35.31$_{+2.89}$

As shown in the above table, under the rigorous online KB evolution settings, the refinement actions generated from DeepRefine can still benefit future queries in most cases. We have added these experiments into Appendix E of our revision.

AuthorsOpenReview ↗

Response to Reviewer 4ocp (3/3)

Follows up to Response to Reviewer 4ocp (2/3).

W4 & Q4: Improved answer F1 does not directly establish edit correctness.

Thank you for pointing out that. To provide direct evidence, we conducted a source-linked audit using the same matched query–source pairs for DeepRefine-4B and DeepRefine-8B. Specifically, we sampled 30 shared queries from each of NQ, PopQA, 2WikiMultiHopQA, MuSiQue, and LoCoMo. For every sampled query, we independently evaluated the edit produced by DeepRefine-4B and DeepRefine-8B against the same linked source documents.

We counted an edit as justified only when the source explicitly stated or unambiguously entailed the complete subject–relation–object claim. We additionally checked relation direction and whether the source fact was attributed to the correct entity. The per-dataset results are shown below.

DatasetSamplesDeepRefine-4B justifiedDeepRefine-4B rateDeepRefine-8B justifiedDeepRefine-8B rate
NQ302790.0%2790.0%
PopQA302893.3%2686.7%
2WikiMultiHopQA302996.7%2686.7%
MuSiQue302790.0%2893.3%
LoCoMo302583.3%2583.3%
Average--90.7%-88.0%

Averaged across the five datasets, the source-justification rates were 90.7% for 4B and 88.0% for 8B. Both models produce source-supported edits for the large majority of the audited queries across all five datasets. The remaining errors fell into two main categories. First, some edits linked two entities that both appeared in the source but assigned a relation that the source did not support. Second, some edits transferred a valid source fact to the wrong entity.

Dear Reviewer 4ocp,

As the discussion deadline is approaching, we wanted to kindly ask whether our response has addressed your concerns. We are happy to provide any further clarification before the discussion closes.

Thank you again for your time.

Reviewer_b9TyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DeepRefine, an RL-trained agent that refines an existing knowledge graph or LLM-Wiki style KB using incoming user queries. For queries judged unanswerable from the initially retrieved subgraph, the agent expands the subgraph over several judgement turns, diagnoses likely defects, and applies insert_edge / delete_edge / replace_node edits to the shared KB. The Qwen3-4B / 8B policy is trained with GRPO on HotpotQA using a "Gain-Beyond-Draft" reward: answer F1 on the refined KB minus answer F1 on the draft KB. Across five QA datasets, three KB constructors and four retrievers, the paper reports average F1 gains of 0.5–5.6 points, together with ablations, a held-out-query test, a time comparison with full reconstruction, and a synthetic defect benchmark.

Critical strengths
  1. A practical problem setting. Incrementally repairing a KB that is already deployed, driven by the queries it actually receives, avoids retraining the constructor and rebuilding the whole KB. In the tested settings, refinement takes far less time than full reconstruction (Table 2). The three edit operators are simple and backend-agnostic.

  2. Broad and directionally consistent gains. In Table 1, 113 of 120 dataset-level results improve over the unrefined KB, and all 24 average gains are positive (0.54–5.60 F1) across three constructors, four retrievers and two model sizes. The reported averages are internally consistent. Gains are largest for triple-based retrievers, which matches the paper's explanation that edits act on triples rather than text.

  3. Controlled evidence that RL improves the prompted pipeline. The trained policy beats the same pipeline without RL on all ten Table 3 comparisons (+0.77 / +1.27 average F1 for 4B / 8B). On the synthetic defect benchmark, DeepRefine-8B exceeds prompted Qwen3-8B by 4.4, 8.3 and 6.1 points for incompleteness, incorrectness and redundancy (Figure 7). The held-out test (Appendix E) shows that edits also help queries that did not drive them.

Critical weaknesses

Anonymity violation in the supplementary material (decisive for my rating). The supplementary material is not properly anonymised. In both released training files, deeprefine_data/hotpotqa_train_refinement_gbd_simple.parquet and deeprefine_data/hotpotqa_valid_refinement_gbd_simple.parquet, every row (662 in total) stores an unmasked absolute path in the field extra_info.interaction_kwargs.full_graph_data_path. This path contains what appears to be a personal name used as a username. Elsewhere in the released code, the same path prefix is masked as /data/***/ (e.g. autorefiner/scripts/prepare_rl_data.py:50), so this occurrence was evidently missed during anonymisation. The ICLR 2027 Author Guidelines treat revealing author identity in the supplementary material as grounds for desk rejection. I have raised this with the AC in a separate comment, and my rating reflects it. The three technical weaknesses below are independent of this issue.

  1. The claimed GBD reward adds no signal under the stated objective, and the released reward differs. In Algorithm 2, the draft score is computed once per query and subtracted from every rollout in its group. For nonzero-variance groups, with trajectories and other objective terms fixed, this shared subtraction leaves the GRPO advantages of Eq. 9 unchanged. In the stated formulation, GBD therefore adds no signal beyond plain refined-KB F1, although it is presented as a contribution. The released gbd_f1_reward.py computes something else. The 4B/8B scripts configure binary substring/LLM-judge accuracy with quadrant gains of (0, 1, −0.3, 0.2), plus a format penalty, and the docstring cites a "Figure 13" that is not in the paper. For fixed draft correctness, the gain component alone is again a positive affine transform of refined accuracy. Whether the complete objective is equivalent also depends on the format penalty, KL placement and normalisation details. Finally, the released rollout generates reward answers with a generator configured as deepseek-v3, whereas Appendix A names Qwen2.5-7B as the reader.

  2. The source of the gains and the evaluation protocol need separating. RL adds 0.77 / 1.27 points on top of a prompted pipeline that already gains about 3.2–3.6, and no variance is reported despite "averaged under 3 times". The refiner also reads up to 15 linked source passages, yet there is no baseline that simply re-extracts triples from those passages for the same queries. Refinement queries are chosen from the evaluation set itself, and the held-out selector still sees the complement. Suppose Tables 1 and 5 use matched refinement runs, the same query set and scoring, and equal per-query weighting. Their reported means then imply lower selected-query F1 in three of the four MuSiQue configurations; for Subgraph-4B, the implied mean falls from 14.26 to 10.03 while the held-out mean rises. These are conditional reconstructions that need confirming, with both subsets reported directly.

  3. The reported results cannot yet be traced to the released artefacts, and the synthetic benchmark protocol needs clarification. Appendix A states group size 16, LR 1e-6 and 40 steps for 4B. The released evaluation script references an 8B checkpoint whose name, under the training script's naming template, suggests group size 6 and LR 1e-7, and a commented-out 4B reference specifies step 50. These references do not establish which checkpoints or settings produced the tables. The released training files have 531 / 131 rows, versus the stated 5,000-sample 8:2 split. They also lack the draft_kg field, without which the released training entry point stops with an error. The smallkg_chunk_genacc_5000 files referenced by the training scripts are absent, so which data produced the reported checkpoints remains unverified. Finally, Appendix C retains corrupted examples with F1 below 0.6, whereas Figure 7 reports a redundancy baseline mean of 64.23. These cannot both describe the same per-example scores on the same retained samples.

Questions for authors
  1. Objective: Which reward, reader and correctness judge produced the reported 4B and 8B models? Under group-normalised GRPO, what do the draft term and the quadrant asymmetry contribute? If the implemented reward differs meaningfully from plain answer scoring, please ablate it; otherwise please correct the GBD claim.

  2. Uncertainty: What varied across the three repetitions? Please report the spread for Tables 1 and 3, and a paired comparison of the full model against "w/o RL".

  3. Protocol: Do Tables 1 and 5 use the same refined KB and scoring? Please report Table 1 gains separately for the selected and held-out queries. How does DeepRefine compare with directly re-extracting triples from the same linked passages for the same queries?

  4. Reproducibility: Please give the resolved hyperparameters, step counts and training data for the reported checkpoints, and explain how the Figure 7 redundancy baseline relates to the F1 < 0.6 filter.

My rating of 1 (clear rejection) is based on the anonymity violation described at the start of the weaknesses. On technical grounds alone, my assessment would be weak rejection. I would reconsider that technical assessment, potentially to weak acceptance, if the authors identify the actual objective and either correct the GBD claim or show what the implemented reward adds, report uncertainty showing that the RL increment exceeds run-to-run variation, and clarify the evaluation protocols by reporting selected and held-out results directly. Subset heterogeneity or a lack of statistical significance would not, on its own, be decisive.

9 comments
AuthorsOpenReview ↗

Response to Reviewer b9Ty (1/4)

We appreciate the insightful feedback from Reviewer b9Ty for highlighting our work’s practical problem setting, method performance gain and effectiveness of RL training!

Regarding anonymity in the supplementary material.

We thank the reviewer for bringing this issue to our attention. We sincerely apologize for overlooking the absolute path stored in the extra_info.interaction_kwargs.full_graph_data_path field of the two Parquet files. We had proactively anonymized directly visible text and masked the corresponding path in the source code. The remaining occurrence was an unintentional omission, not an attempt to disclose our identity.

Because Parquet files usually can not be directly inspected without a specific parser or code, we unfortunately missed this field during our anonymization check. We should have checked the supplementary materials more thoroughly. However, we do not believe that an account name in a path, by itself, establishes the identity of an author, although we understand why its presence raises a concern.

Consequently, we respectfully ask the reviewer and the Area Chair to consider that this was an inadvertent omission in the context of our broader anonymization efforts. We have submitted a thoroughly checked and anonymized version of the supplementary material. Thanks again!

AuthorsOpenReview ↗

Response to Reviewer b9Ty (2/4)

Follows up to Response to Reviewer b9Ty (1/4).

The claimed GBD reward adds no signal under the stated objective, and the released reward differs

We thank the reviewer for this careful observation. We agree with the reviewer’s mathematical point, which means GBD produces the same advantages as refined-answer F1 score alone. However, we would like to clarify that in our implementations, the draft F1 scores are recomputed per rollout and could be different in a same group. We apologize for the confusion in Algorithm 2 and initially released code, and we have uploaded the updated code and fixed that in our revision.

To be specific, different global random seeds are used for RAG-based reward calculating across rollouts in a group to expose training to multiple realizations of RAG randomness, rather than relying on a single fixed global seed. Within each rollout, the draft and refined evaluations shared a global seed to facilitate a paired comparison. This design was motivated by the goal of making refinement-gain measurement less dependent on a particular RAG realization and randomness. Beyond its effect on optimization, compared with absolute scores, GBD provides an explicit and interpretable measure of the observed performance gain or degradation relative to the draft KB. We will clarify this distinction in the revision.

To further demonstrate that varying the global random seeds of the RAG processes used for reward computation across different rollouts within the same group can lead to different F1 scores, we evaluate 50 examples selected from the training data. For each example, we keep the draft KB and all model and retrieval configurations fixed, using Qwen3-Embedding-0.6B\texttt{Qwen3-Embedding-0.6B} as the retriever, Qwen2.5-7B-Instruct\texttt{Qwen2.5-7B-Instruct} as the reader, top-10 retrieval, and a decoding temperature of 0. For robustness, we repeat the evaluation using three independently sampled, non-overlapping random-seed groups, each containing 12 global seeds. As shown in the following table, between 9 and 11 out of 50 examples produce different F1 scores across seeds, corresponding to variation rates of 18%-22% and an average of 19.3%.

Seed GroupSeeds per ExampleSame F1Different F1Difference Rate
Group 112391122%
Group 21241918%
Group 31241918%
Average1240.39.719.3%

We also trained a model DeepRefine-4B w/ F1 Reward and evaluated it on 5 benchmarks under the naive constructor and two triple retrievers to demonstrate the gains brought from GBD reward design.

VariantsRetrieversNQPopQA2WikiQAMusiqueLOCOMOOverall
-Subgraph26.4354.4833.5313.8225.8630.82
DeepRefine-4B w/ GBD RewardSubgraph29.5560.0936.0415.9329.6434.25$_{+3.43}$
DeepRefine-4B w/ F1 RewardSubgraph28.1057.7535.3613.6527.9032.55$_{+1.73}$
-ToG25.5554.9243.7418.2122.6433.01
DeepRefine-4B w/ GBD RewardToG29.2159.2749.1625.7324.5337.58+4.57_{+4.57}
DeepRefine-4B w/ F1 RewardToG28.3754.8940.3120.0520.1432.75$_{-0.26}$

As the table shows, the performance of DeepRefine trained with F1 score as reward is obviously lower than DeepRefine trained with GBD reward. And interestingly, DeepRefine w/ F1 reward perform very bad on ToG retriever. It may because our RAG evaluation in reward calculations uses the Subgraph retriever and with F1 score as the reward, the policy only learns to optimize its performance under the randomness of Subgraph retriever. However, with GBD reward, the rewards are exactly from the gains, will not be influenced very much by the randomness of the RAG process during the reward calculations, which is a more stable design than F1 reward. We have added this result in Appendix F of our revision.

We also sincerely apologize for the discrepancies between the paper and the initially released supplementary code. The supplementary package contained an outdated v1 implementation. We have uploaded the updated code.

AuthorsOpenReview ↗

Response to Reviewer b9Ty (3/4)

Follows up to Response to Reviewer b9Ty (2/4).

Uncertainty: What varied across the three repetitions? Please report the spread for Tables 1 and 3, and a paired comparison of the full model against "w/o RL".

There are no explicit differences between the three repeated implementations, except for the floating point errors that would caused by batched inference[1][2] or other reasons, which would lead to varied RAG results and is the reason why we do this. We did not report them because we found the variances are much smaller than the gains. To support this, we demonstrate the variances of the RL performance gains under 3 times' implementations of Table 3 in the following table. Because the results of Table 1 not only have already demonstrated strong performance gains, but also align with the results in AutoGraph-R1. As the following results suggest, the standard variances are relatively small in most cases and the performance gains brought by RL are still statistically significant especially in multi-hop benchmarks. We have also added that in Table 3 of our revision.

VariantsNQPopQA2WikiQAMuSiQueLoCoMoOverall Avg.
DeepRefine-4B29.38 (±0.15)59.68 (±0.28)42.60 (±0.18)20.83(±0.20)27.09(±0.04)35.92 (±0.11)
DeepRefine-4B w/o RL28.74 (±0.03)58.89 (±0.13)41.37 (±0.50)19.86 (±0.13)26.89 (±0.12)35.15 (±0.13)
DeepRefine-8B30.29 (±0.19)59.93 (±0.05)44.77 (±0.59)21.71 (±0.16)27.02 (±0.26)36.74 (±0.12)
DeepRefine-8B w/o RL29.19 (±0.17)59.73 (±0.13)42.71 (±0.31)20.62 (±0.12)25.08 (±0.14)35.47 (±0.15)

[1] PyTorch, “Numerical accuracy. https://docs.pytorch.org/docs/2.15/notes/numerical_accuracy.html

[2] Yuan J, Li H, Ding X, et al. Understanding and mitigating numerical sources of nondeterminism in llm inference[J]. Advances in Neural Information Processing Systems, 2026, 38: 169819-169851.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/3)

Dear Reviewers and AC,

We appreciate very much your time and constructive suggestions on our work!

We would like to provide a summary of the strengths of our work commonly raised by the reviewers, as well as some key points which clarify the common concerns of the reviewers.

Through the rebuttal, we have addressed the reviewers' concerns in details and actively engaged in discussion with the reviewers, while reviewers b9Ty and AVPX have both raised their scores. The increased scores are listed below:

  • b9Ty: 1->2.
  • AVPX: 2->3.

Thank the reviewers again for your recognitions and highlighting our work’s contributions! The common strengths are listed below:

  • DeepRefine addresses a practical, timely, and well-motivated problem: post-construction, query-driven incremental refinement of already deployed KBs avoids expensive full reconstruction and retraining while taking substantially less time. (b9Ty, 4ocp, AVPX, ycWh)
  • The proposed agentic refinement framework is reasonable, clearly structured, and easy to follow, cleanly separating multi-turn answerability judgment, error abduction, and refinement action generation (4ocp, AVPX), while addressing the lack of golden refinement trajectories via RL from query–answer pairs (ycWh).
  • Extensive evaluations across five QA datasets, three KB constructors, and four retrievers demonstrate broad and directionally consistent downstream gains, while held-out query evaluations provide clear evidence of cross-query transfer beyond the refinement-driving queries. (b9Ty, 4ocp, AVPX)
  • Controlled ablations and the synthetic defect benchmark provide solid evidence that RL optimization and error abduction consistently improve over the base model across incompleteness, incorrectness, and redundancy defects. (b9Ty, AVPX)
AuthorsOpenReview ↗

General Response (2/3)

Follows up to General Response (1/3)

Some key clarifications and improvements to our work based on the reviewers' concerns are listed below: (Changes in the revision are marked in blue)

  • Clarification and ablation of the Gain-Beyond-Draft (GBD) reward under GRPO. (b9Ty, 4ocp, AVPX)

    • We clarified that in our implementation, draft answers are recomputed per rollout coroutine under paired request-level seeds shared between draft and refined READER calls, while varying across rollouts in a group (yielding an average F1 variation rate of 19.3% in our seed diagnostic). Furthermore, we trained a direct ablation variant DeepRefine-4B w/ F1 Reward. DeepRefine trained with GBD substantially outperforms the refined-F1 variant across all 10 dataset–retriever settings (+1.70 average F1 on Subgraph and +4.83 on ToG, +3.27 overall), demonstrating the effectiveness of paired, relative credit assignment over noisy RAG-dependent absolute F1 scores.
    • [Changes in the paper]: We have updated Algorithm 2 in Appendix D, and added the reward function comparisons, training dynamics (Figure 8), and benchmark results (Table 9) in Appendix F.
  • Rigorous online historical-to-future evaluation and selected vs. non-selected query decomposition. (b9Ty, 4ocp, AVPX)

    • We conducted a strict historical-to-future evaluation where benchmark queries are split before coverage selection (500 historical queries for selection and refinement; 500 strictly unseen future queries solely for evaluation). DeepRefine still consistently improves strictly unseen future queries (+1.94 to +2.89 average F1) while yielding even larger gains on historical queries. We also decomposed Table 1 into selected and non-selected queries: DeepRefine improves all 24/24 configurations on selected queries (+5.41 overall average F1) and 23/24 configurations on non-selected queries (+1.49 overall average F1).
    • [Changes in the paper]: We have added the selected vs. non-selected summary (Table 2) and discussions in Section 5.2, the rigorous historical-to-future online evaluation (Tables 6 and 7) in Appendix E.1, and the full per-dataset selected/non-selected results (Tables 8 and 10) in Appendix E.2.
  • Factual correctness and source-support audit of generated KB edits. (4ocp, AVPX)

    • We conducted a strict source-linked audit over 150 matched query–source pairs across all five datasets (30 queries per dataset), counting an edit as justified only when the linked source explicitly stated or unambiguously entailed the complete subject–relation–object claim with correct direction and entity attribution. DeepRefine-4B and DeepRefine-8B achieve 90.7% (136/150) and 88.0% (132/150) source-justification rates, respectively, confirming that the vast majority of persistent KB edits are grounded in source evidence.
  • Research gap and positioning relative to KG refinement, iterative KG editing, and agent memory methods. (AVPX, ycWh)

    • We clarified the core research gap: existing KG-editing and completion methods (e.g., TRAIL, KG-BERT) primarily add new entities or relations, whereas disambiguation methods (e.g., NeuSymEA) focus on correcting or simplifying knowledge. This leaves open a general refinement framework that jointly addresses incompleteness, incorrectness, and redundancy through various operations.
    • [Changes in the paper]: We have sharpened the research gap in Abstract, Section 1 and expanded the related work discussions contrasting TRAIL, GraphRefine, and KGER in Section 2.3.
  • Matched-budget comparison with non-persistent retrieval-and-reasoning (ReAct). (AVPX)

    • We further implemented a matched-budget ReAct w/o KB Edit baseline under the same retrievers, reader LLM, and context budget without persisting any edits to the KB. DeepRefine-8B achieves higher overall average F1 on both Subgraph and ToG, demonstrating that inference-time search/reasoning alone cannot resolve underlying KB defects (incompleteness, incorrectness, and redundancy) as persistent KB refinement does.
    • [Changes in the paper]: We have added the implementation details, results (Table 11), and discussions of the matched-budget ReAct comparison in Appendix G.
AuthorsOpenReview ↗

General Response (3/3)

Follows up to General Response (2/3)

  • Run-to-run variance, reproducibility, and supplementary material updates. (b9Ty)

    • We reported standard deviations across 3 repetitions for our ablation study, showing that run-to-run variations are small relative to the RL gains, especially on multi-hop benchmarks. We also sincerely apologized for the unintentional path omission in the v1 Parquet metadata and uploaded a thoroughly checked, anonymized, and version-matched code. We also believe that only an account name in a path, by itself, can not establish the identity of an author.
    • [Changes in the paper]: We have added standard deviations across 3 runs in Table 4 (Section 5.3).
  • Other changes in the paper.

    • The policy's access to entity-linked source text chunks (CGq(L)\mathcal{C}_{G_q^{(L)}}) during refinement action generation has been explicitly formulated and clarified in Section 4.1 (Eq. 7), Section 4.3, and Appendix J (Figure 13). (b9Ty, 4ocp)
    • The retrievers used across stages of the synthetic defect benchmark (the simple triple retriever for the F1>0.95\text{F1} > 0.95 and F1<0.6\text{F1} < 0.6 filtering stages, and the Subgraph retriever for final evaluation) have been explicitly specified in Appendix C and Section 5.3 (Figure 7). (b9Ty)

We remain open to further discussions to resolve any remaining valuable concerns or address additional questions. Thank you for reviewing our paper again!

Sincerely,

Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.