acceptodds
Under review as a conference paper at ICLR 2027

Preselection-Free Knowledge-Grounded Reinforcement Learning

Abstract

Existing agents solve sequential decision-making tasks by accessing knowledge, such as expert-designed data, or navigating the environments through trial and error. Research has explored knowledge as supervision or reinforcement learning for interactive improvement, but typically studied independently. Recent work introduces a unified paradigm called Knowledge-Grounded Reinforcement Learning (KGRL), which studies how RL can efficiently leverage external knowledge and how integrating these two complementary signals can enhance learning. However, prior KGRL methods require an expert-preselected knowledge set for each task, which is hard to obtain in real-world applications. Without it, performance can degrade significantly due to misleading information. We propose a trade-and-curate mechanism for KGRL that trades stochastic information among knowledge policies and curates knowledge-policy fusion, making the agent resilient to unhelpful information in an unselected knowledge set. Our experiments in KGRL MiniGrid environments show that the trade-and-curate mechanism achieves up to 89.6% performance improvement without knowledge preselection.

Primary area:
reinforcement learning
Keywords:
deep reinforcement learning, knowledge-grounded reinforcement learning, policy fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RRkLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript addresses the critical limitation in KGRL regarding the reliance on human preselection of task-relevant knowledge policies, and proposes a preselection-free method named TC-KIAN based on the KIAN framework. The proposed method introduces an auxiliary loss based on the pairwise KL divergence between policies to regularize knowledge representations, alongside a policy fusion mechanism that utilizes query and key norms to modulate fusion weights and distribution temperatures. The manuscript achieves superior performance over baselines including KIAN across 12 MiniGrid tasks as well as stress tests with redundant and uniform knowledge policies. I find that the manuscript possesses certain research value, but it still requires essential theoretical clarifications and controlled ablation experiments to substantiate its core claims.

Critical strengths
  1. The manuscript identifies an important and practically meaningful research problem by focusing on the limitation of existing KGRL methods that rely on human-preselected knowledge, shifting the research focus toward the effective fusion of unselected knowledge sets containing redundancy and distraction. The motivation is clear and sound.
  2. Explicitly incorporating behavioral similarity between policies into knowledge representation learning is conceptually sound. This design mitigates the complete reliance of knowledge representation learning on sparse environmental rewards to a certain extent.
  3. The engineering motivation behind Curated Policy Fusion is intuitive, as it identifies the limitation of the normalized inner product in the original KIAN where attention weights become diluted as the knowledge repository grows. Furthermore, it demonstrates good compatibility with existing frameworks.
  4. The experimental evaluation not only includes primary performance comparisons, but also designs stress tests with duplicated and uniform policies, ablation studies on the two core modules, and visualizations of attention weights and representations. These experiments provide a multifaceted empirical analysis.
Critical weaknesses
  1. The core auxiliary loss, Trade Loss, may be mathematically unbounded from below, and the theoretical assumptions in the appendix diverge significantly from the actual algorithm. In Equations (3) to (5), without explicit constraints on the norms of knowledge keys or pairwise distances, when there exist policy pairs whose KL divergence exceeds the mean threshold thth, the repulsive term can drive Ltrade→−∞\mathcal{L}_{\mathrm{trade}} \to -\infty by infinitely scaling up the norms of the keys. In addition, Appendix A.1 assumes that the attractive and repulsive terms share a common finite optimal solution, which rarely holds in an unconstrained space. In my view, for an objective function that could be unbounded from below, the continuous decrease of the loss observed in Figure 5c does not naturally equate to a stable improvement in representation quality.
  2. The empirical attribution of the mechanism is not entirely clean, as the gains from simple attention temperature adjustments have not been fully ruled out. The original KIAN employs an inner product of unit vectors where logits are constrained to [−1,1][-1, 1], which limits the theoretical maximum attention weight of a single useful policy to only about 6.88%6.88\% under the stress test with 100 distracting policies. TC-KIAN preserves the query norm, which essentially introduces a learnable inverse temperature parameter, while Equation (7) also utilizes the key norm to adjust the temperature of the action distributions of knowledge policies. Although Figure 7 provides module ablations, critical and carefully controlled baselines are missing. In my opinion, the current experiments do not sufficiently prove whether the performance improvement stems from sophisticated knowledge representation learning or merely from overcoming the restricted temperature limitation of the original KIAN.
  3. I find that the claim of being preselection-free is overstated, because the current experiments hardly support the generalizability of this conclusion under realistic and complex knowledge repositories. The existing evaluations are conducted exclusively on eight handcrafted, isomorphic rule based policies designed for MiniGrid. The stress tests merely perform exact duplicate sampling of these eight policies or inject uninformative uniform random policies. This setup is far removed from real world scenarios that feature contaminated knowledge with high-confidence errors, partially detrimental states, or conflicting policy objectives.
  4. The arguments regarding computational scalability and sparse selection are not sufficiently rigorous. Trade Loss requires computing the pairwise KL divergence across all policy pairs, which incurs a quadratic complexity of O(n2∣A∣)O(n^2\vert{}\mathcal{A}\vert{}) with respect to the number of knowledge policies nn. Meanwhile, Equations (6) and (7) still perform dense forward passes and weighted summation across all knowledge policies, and concentrated attention weights do not equate to achieving computational sparse selection or automated pruning. Furthermore, the pruning experiments in Appendix G rely on manually selecting representative policies from three categories, namely PK, OD, and G, which fails to demonstrate that the model can automatically identify the optimal subset during inference.
  5. The manuscript also exhibits weaknesses regarding statistical evidence and baseline configurations. Specifically, several learning curves in Figure 3 and Figure 7 exhibit high variance with noticeably overlapping confidence intervals. Moreover, the calculation methodology for the claimed 89.6% performance gain remains ambiguous. Crucial implementation details are also missing, such as whether the pairwise policy loss is summed or averaged across pairs.
Questions for authors
  1. I would like the authors to clearly explain how the unbounded growth of the key norm is prevented during actual training. Could you provide the empirical trajectory of the knowledge key norms during training, and explain how the theoretical assumptions in Appendix A can be revised to align with the practical algorithm?
  2. Could the authors provide a direct comparative experiment with KIAN plus a learnable or tuned temperature, as well as ablation results where the query norm and key norm are individually fixed, to demonstrate the independent contribution of the Trade Mechanism over simple temperature adjustment?
  3. Please provide numerical tables corresponding to the core experiments, along with the comparative results for TC-KIAN under an expert-selected knowledge set.
  4. Please clarify the normalization scheme for the summation over policy pairs in Equation (3), as well as the numerical smoothing details when computing the KL divergence for deterministic rule-based policies.
9 comments
AuthorsOpenReview ↗

Author Response - Overview

We thank the reviewer for their insightful comments! Please see our response below. We uploaded a revised manuscript that includes the additional experimental results (starting on page 21).

Connection between Theory and the Practical Algorithm
Please see [Part 1] below for the detailed response and additional experimental results.

Comparison with Temperature Tuning
Please see [Part 2] below for the detailed response and additional experimental results.

Stress Tests on Structurally Misleading Knowledge
Please see [Part 3] below for the detailed response and additional experimental results.

Computational Scalability and Sparse Selection

“The arguments regarding computational scalability and sparse selection are not sufficiently rigorous. …, and concentrated attention weights do not equate to achieving computational sparse selection or automated pruning. Furthermore, the pruning experiments in Appendix G rely on manually selecting representative policies from three categories, namely PK, OD, and G, which fails to demonstrate that the model can automatically identify the optimal subset during inference.”

We DO NOT claim that our method is performing sparse selection or automated pruning. Your understanding of the forward passes is correct, but we never claimed in our paper that our attention weights on knowledge policies are related to sparse selection or automated pruning.

Our computational analysis in Appendix F shows that, even with the additional trade-loss computation, the computational overhead is minimal. This is unrelated to sparse selection.

Our knowledge pruning analysis in Appendix G is an ablation study that validates whether the key representations learned by the trade loss enable robust performance under knowledge pruning during inference. We adopt an autonomous, not manual, pruning method derived from prior work in explainable ML [1]. Our purpose is to analyze the effects of our trade loss, and we DO NOT claim to propose any autonomous knowledge pruning method in this paper.

We believe this comment reflects a major misunderstanding of our work, and we kindly ask the reviewer to reread the paper for clarification.

[1] Yi-Lin Tuan, Connor Pryor, Wenhu Chen, Lise Getoor, and William Yang Wang. Local explanation of dialogue response generation. Advances in Neural Information Processing Systems, 34:404–416, 2021.

Statistical Evidence, Metric for Improvement Percentages, and Implementation Details

“The manuscript also exhibits weaknesses regarding statistical evidence and baseline configurations. Specifically, several learning curves in Figure 3 and Figure 7 exhibit high variance with noticeably overlapping confidence intervals.”

Thank you for pointing this out. We discussed the learning variance in our original limitations section (Appendix H): “TC-KIAN’s learning-curve variance is sometimes higher than that of KIAN, the state-of-the-art KGRL method, showing that while it converges faster and achieves higher rewards, its stability can be improved in future work.”

We also note that most of our results are considered statistically significant improvements since only a small portion of confidence intervals overlap.

“Moreover, the calculation methodology for the claimed 89.6% performance gain remains ambiguous.”
“Please provide numerical tables corresponding to the core experiments, along with the comparative results for TC-KIAN under an expert-selected knowledge set.”

For Figure 3, we provide the table of average rewards and improvement percentages in the revised manuscript (Table 3, page 22).

  • TC-KIAN’s improvement over KIAN is computed as (TC-KIAN (KS-all) - KIAN (KS-all)) / (KIAN (KS-all) - minimum environment reward).
  • KIAN’s performance drop is computed as (KIAN (KS-all) - KIAN (KS-selected)) / (KIAN (KS-selected) - minimum environment reward).
  • TC-KIAN’s performance drop is computed as (TC-KIAN (KS-all) - KIAN (KS-selected)) / (KIAN (KS-selected) - minimum environment reward).

In Figure 4 of the original manuscript, we reported the results of TC-KIAN with an expert-selected knowledge set (KS-preselected).

“Crucial implementation details are also missing, such as whether the pairwise policy loss is summed or averaged across pairs.”

Thank you for pointing this out! We took the average after summing the pairwise trade loss across policy pairs. We will add this implementation detail. Note that whether to take the average in Eq. (3) has limited effect on the learning results, as it merely adds a constant to the function.

“..., as well as the numerical smoothing details when computing the KL divergence for deterministic rule-based policies.”

We clarify that our knowledge policies are all stochastic. Each of them is a probability vector with each component >= 0 and <= 1, and the sum of all components = 1. Therefore, we can naturally compute KL divergence on these probabilities.

AuthorsOpenReview ↗

[Part 1] Author Response: Connection between Theory and the Practical Algorithm

“The core auxiliary loss, Trade Loss, may be mathematically unbounded from below, and the theoretical assumptions in the appendix diverge significantly from the actual algorithm.”
“Could you provide the empirical trajectory of the knowledge key norms during training, and explain how the theoretical assumptions in Appendix A can be revised to align with the practical algorithm?”

Thank you for pointing this out. First, we clarify that only the bounded key norm assumption affects whether the trade loss is bounded below. Other assumptions are added mainly to study non-trivial cases (e.g., minimum separation = 0 implies trade loss = 0) and to perform theoretical analysis on a locally smooth function (Lemmas 2 and 3). This assumption has limited effect on whether the trade loss is bounded. From now on, we will focus on the bounded key norm assumption.

In our experiments, the bounded key norm assumption is never violated (Figure 5c), so whether the key norms are bounded has limited effect on the convergence performance of our current results. We did not include this constraint in our implementation because we want to follow KIAN as closely as possible to disentangle the constraint's effects from others. However, the bounded key norm assumption is accessible and easy to satisfy; for example, we can pass our learnable key vectors through $\alpha \cdot tanh(k_j)$. In this case, the key norm is bounded by $\alpha$.

We evaluated TC-KIAN and TC-KIAN (bounded keys, α\alpha = 10) on Doorkey-5x5 (Figure 13, page 21). The external knowledge sets include KS-all, Extra-stresstest-R, and Extra-stresstest-U. The results show that TC-KIAN with bounded key norms does not affect convergence and, in general, performs better than TC-KIAN without bounded key norms.

“..., the continuous decrease of the loss observed in Figure 5c does not naturally equate to a stable improvement in representation quality.”

We clarify that Figure 5c is NOT meant to show improvement in representation quality. Figures 6 and 8 are more related to that. Figure 5c shows two things:

  • Simultaneously minimizing the trade loss and maximizing the rewards does not lead to unstable knowledge key training. This implies that the two objectives are not contradictory, which is important for stable learning.
  • KIAN also implicitly minimizes the trade loss while seeking better performance. This suggests that explicitly minimizing the same loss in TC-KIAN may explain our performance improvement.
AuthorsOpenReview ↗

[Part 2] Author Response: Comparison with Temperature Tuning

“..., critical and carefully controlled baselines are missing. In my opinion, the current experiments do not sufficiently prove whether the performance improvement stems from sophisticated knowledge representation learning or merely from overcoming the restricted temperature limitation of the original KIAN.”

Actually, in Figure 7, the results for TC-KIAN without trade loss (C-KIAN) show how much the performance improvement depends on knowledge representation learning, since trade loss is the main driver of learning these representations. Without trade loss, the agents perform worse in two challenging environments, MultiRoom-N4-S5 and KeyCorridorS3R3, showing that learning functionality-aligned knowledge representations improves performance.

In addition, our curated policy fusion (Eq. (6)), or C-KIAN, can be seen as automatically tuning the fusion temperature in KIAN (Eq. (1)), which overcomes the restricted temperature limitation in KIAN while obviating the need for manual temperature tuning for each task.

“Could the authors provide a direct comparative experiment with KIAN plus a learnable or tuned temperature, as well as ablation results where the query norm and key norm are individually fixed, to demonstrate the independent contribution of the Trade Mechanism over simple temperature adjustment?”

TC-KIAN w/o trade loss (C-KIAN) in Figure 7 can be seen as KIAN plus a learnable temperature. By directly comparing Eq. (1) and Eq. (6), the only difference is that Eq. (6) has $\| \mathbf{u}_t \|$ in the softmax, which is a learnable value.

We note that KIAN with fixed query and key norms causes entropy imbalance among knowledge policies, leading the agent to always choose the inner policy and ignore the external ones. This issue is detailed and discussed in the original KIAN paper.

We compared TC-KIAN with the suggested variant: KIAN with the fusion temperature in Eq. (1) manually tuned (KIAN-T) (Figure 11, page 21). This will answer whether our trade loss and curated policy fusion, which automatically tunes the fusion temperature in Eq. (6), achieve better performance and reduce the need for manual temperature tuning.

Our experimental setup:

  • The default temperature parameter is 1 (KIAN), and we added the results for temperature = 0.5 (KIAN-T0.5) and 0.1 (KIAN-T0.1).
  • We evaluate KIAN, KIAN-T, TC-KIAN, and TC-KIAN w/o trade loss (C-KIAN) on four tasks: Doorkey-5x5, Dynamics-Obstacle-Random-6x6, MultiRoom-N4S5, KeyCorridor-S3R3.
  • We use KS-all as the external knowledge set, which is not preselected.

In Figure 11, we can observe:

  • In general, TC-KIAN outperforms KIAN-T, except in Dynamics-Obstacle-Random-6x6, where temperature 0.1 converges faster, but the final performance is close to TC-KIAN. This suggests that our trade loss and curated policy fusion, which automatically tunes the fusion temperature, improve performance.
  • No single temperature parameter is universally better than others. For example, temperature = 0.1 performs better in Dynamics-Obstacle-Random-6x6 but completely fails in KeyCorridor-S3R3.
  • KIAN-T can lead to reward curves with higher variance than TC-KIAN. This is especially obvious in Doorkey-5x5 and MultiRoom-N4S5.
  • KIAN-T can outperform C-KIAN in some tasks, implying that KIAN with careful temperature tuning can outperform KIAN with automatically tuned temperature. However, KIAN-T requires manual, task-specific tuning. In addition, TC-KIAN outperforms both methods, showing that our trade loss improves performance while requiring no manual temperature tuning.

We highlight that one of our main contributions is identifying the issue with a non-tuned fusion temperature in KIAN and proposing an automatic tuning method (our curated policy fusion) to address it. The original KIAN paper does not discuss that fusion temperature can affect performance, nor does it tune that parameter in its implementation. For a fair comparison, we follow their implementation closely and report KIAN results with temperature = 1.

Figures 7 and 11 show that adding our trade loss benefits performance, and this is not the only benefit. By learning correct relationships between knowledge representations, we can obtain more interpretable representations, as shown in Figures 6 and 8.

Reviewer_CwTUOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the performance degradation of KIAN when the external knowledge set contains irrelevant or duplicated policies, and proposes TC-KIAN. TC-KIAN uses a trade loss to learn behavioral relationships among policy representations and adjusts the fusion temperature to improve knowledge selection. MiniGrid experiments show that, under the tested settings, the method can improve sample efficiency and final return.

Critical strengths
  • The problem is practically meaningful. Reducing the need for per-task manual knowledge selection can help improve the utilization efficiency of existing policy libraries.
  • The method design is reasonable. Using behavioral differences among policies to provide an auxiliary representation-learning signal, combined with more flexible fusion weights, has a clear design rationale.
  • The experimental evaluation is reasonably broad within MiniGrid. It covers 12 environments and includes stress tests, component ablations, and computational-overhead analysis; results are reported with 5 random seeds and 95% confidence intervals.
Critical weaknesses
  • The claim about prior knowledge preselection is too broad. KIAN has already studied the impact of irrelevant policies, and KIAN-ACE uses a shared initial knowledge set that can contain task-irrelevant policies. The paper should therefore distinguish its robustness contribution more clearly, rather than broadly claiming that prior methods require task-specific expert preselection.
  • Insufficient connection between theory and the practical algorithm. The trade-loss convergence analysis relies on conditions such as bounded key norms and a minimum separation, but it is not explained how these conditions are maintained during training; the compositionality analysis is also insufficient to demonstrate that the learned representations can be reused across tasks.
  • The experiments do not yet adequately isolate the sources and limits of the gains. The existing ablations suggest complementary effects of the components, but comparisons with systematically tuned temperature controls are missing. The stress tests mainly use duplicated and uniform policies, providing limited evidence of robustness to systematically misleading policies.
Questions for authors
  • How large is the additional benefit of the full method relative to simple temperature adjustment? Figure 7 already includes a no-trade-loss ablation that retains curated policy fusion. Beyond this ablation, please compare TC-KIAN with KIAN using a tuned fixed temperature or a learnable global temperature for the fusion weights, and report performance and variability across seeds on representative tasks.
  • How are the assumptions of the convergence analysis satisfied in actual training? Please state whether constraints on key norms and minimum separation are imposed, and clarify whether the batch-mean threshold alone guarantees that the trade loss is bounded below. If the current algorithm does not satisfy the theoretical assumptions, please explicitly delimit the scope of the convergence claim.
  • Can the method handle systematically misleading external policies? Please include policies that output incorrect actions in some states, especially policies that fail at critical decision points, and test different interference ratios. For these additional tests, report absolute returns and confidence intervals. For the existing improvement percentages, specify the metric, reference baseline, and calculation formula.
4 comments
AuthorsOpenReview ↗

Author Response - Overview

We thank the reviewer for their insightful comments! Please see our response below. We uploaded a revised manuscript that includes the additional experimental results (starting on page 21).

Connection between Theory and the Practical Algorithm
Please see [Part 1] below for the detailed response and additional experimental results.

Comparison with Temperature Tuning
Please see [Part 2] below for the detailed response and additional experimental results.

Stress Tests on Structurally Misleading Knowledge
Please see [Part 3] below for the detailed response and additional experimental results.

Claim about Prior Knowledge Preselection

“The claim about prior knowledge preselection is too broad. KIAN has already studied the impact of irrelevant policies, and KIAN-ACE uses a shared initial knowledge set that can contain task-irrelevant policies.”

Thank you for pointing this out. First, we clarify that our statement, “prior KGRL methods require an expert-preselected knowledge set for each task”, means that “prior KGRL methods require an expert-preselected knowledge set for each task to achieve the good performance they reported in their papers”. We will revise our manuscript to include this.

Theoretically, both KIAN and KIAN-ACE can include irrelevant knowledge during training. Still, neither is designed for this, and neither shows stable, strong performance when the knowledge set contains truly irrelevant policies.

KIAN’s paper [1] reports the main results for each task using an expert-preselected, task-specific knowledge set. In their Appendix F, they discuss the effects of adding irrelevant knowledge for one simple task (Unlock), without specifying which irrelevant knowledge policies are included. We believe their work has not sufficiently studied the impact of irrelevant policies on KIAN.

KIAN-ACE’s paper [2] uses the same knowledge set across all tasks, which can contain irrelevant policies for each task. However, their knowledge set contains only three highly similar policies, which all guide a robot arm to move toward something, either the goal, an object, or the other robot arm. We consider these policies to be “suboptimal” rather than “irrelevant” for each task. Therefore, we believe their work has not sufficiently studied the impact of a larger set of diverse, irrelevant policies on KIAN-ACE.

In our work, we have shown that KIAN and KIAN-ACE’s performance degrades significantly under an un-preselected knowledge set (Figure 3). This finding is consistent with the preliminary results discussed in KIAN’s paper and is systematically verified in our work. We thus conclude that these methods are not specifically designed for an un-preselected knowledge set.

[1] Chiu, Z. Y., Tuan, Y. L., Wang, W. Y., & Yip, M. (2023). Flexible attention-based multi-policy fusion for efficient deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 13590-13612.
[2] Ho, Y. J., Chiu, Z. Y., Zhi, Y., & Yip, M. C. (2025). Surgirl: Toward life-long learning for surgical automation by incremental reinforcement learning. IEEE Robotics and Automation Letters, 10(12), 13145-13152.

Metric for Improvement Percentages

“For the existing improvement percentages, specify the metric, reference baseline, and calculation formula.”

For Figure 3, we provide the table of average rewards and improvement percentages in the revised manuscript (Table 3, page 22).

  • TC-KIAN’s improvement over KIAN is computed as (TC-KIAN (KS-all) - KIAN (KS-all)) / (KIAN (KS-all) - minimum environment reward).
  • KIAN’s performance drop is computed as (KIAN (KS-all) - KIAN (KS-selected)) / (KIAN (KS-selected) - minimum environment reward).
  • TC-KIAN’s performance drop is computed as (TC-KIAN (KS-all) - KIAN (KS-selected)) / (KIAN (KS-selected) - minimum environment reward).
AuthorsOpenReview ↗

[Part 1] Author Response: Connection between Theory and the Practical Algorithm

“The trade-loss convergence analysis relies on conditions such as bounded key norms and a minimum separation, but it is not explained how these conditions are maintained during training; …”
“..., and clarify whether the batch-mean threshold alone guarantees that the trade loss is bounded below.”

Thank you for pointing this out. First, we clarify that only the bounded key norm assumption affects whether the trade loss is bounded below. The minimum separation assumption is added mainly to study non-trivial cases, since separation = 0 implies trade loss = 0, and to perform theoretical analysis on a locally smooth function (Lemmas 2 and 3). This assumption has limited effect on whether the trade loss is bounded.

The batch-mean threshold is also not related to the boundedness of trade loss. It provides a pivot point for when two knowledge keys should be pulled together or pushed apart, as explained in Section 4.1. From now on, we will focus on the bounded key norm assumption.

In our experiments, the bounded key norm assumption is never violated (Figure 5c), so whether the key norms are bounded has limited effect on the convergence performance of our current results. We did not include this constraint in our implementation because we want to follow KIAN as closely as possible to disentangle the constraint's effects from others. However, the bounded key norm assumption is accessible and easy to satisfy; for example, we can pass our learnable key vectors through $\alpha \cdot tanh(k_j)$. In this case, the key norm is bounded by α\alpha.

We evaluated TC-KIAN and TC-KIAN (bounded keys, $\alpha$ = 10) on Doorkey-5x5 (Figure 13, page 21). The external knowledge sets include KS-all, Extra-stresstest-R, and Extra-stresstest-U. The results show that TC-KIAN with bounded key norms does not affect convergence and, in general, performs better than TC-KIAN without bounded key norms.

“...; the compositionality analysis is also insufficient to demonstrate that the learned representations can be reused across tasks.”

The compositionality analysis in Property 2 is NOT a direct demonstration of representation reuse across tasks. Property 2 shows that the learned knowledge representations have semantic relationships similar to their corresponding knowledge policies. Preserving these semantic relationships is essential for the knowledge representations to be non-training-specific. That is, the same group of knowledge policies used in two training sessions should not have two sets of representations with completely different internal relationships among knowledge.

Knowledge reuse in learning new tasks is out of scope for this work, so we do not discuss this specifically. But in the original KIAN paper, they mention that, with learned knowledge representations, an agent does not need to re-navigate the entire knowledge set when learning new tasks. This will require learning the correct relationships between knowledge representations, which is exactly what our trade loss is designed for.

AuthorsOpenReview ↗

[Part 2] Author Response: Comparison with Temperature Tuning

“The existing ablations suggest complementary effects of the components, but comparisons with systematically tuned temperature controls are missing.”
“..., please compare TC-KIAN with KIAN using a tuned fixed temperature or a learnable global temperature for the fusion weights, ...”

We note that TC-KIAN w/o trade loss (C-KIAN) in Figure 7 can be seen as KIAN plus a learnable temperature. By directly comparing Eq. (1) and Eq. (6), the only difference is that Eq. (6) has $\| \mathbf{u}_t \|$ in the softmax, which is a learnable value. C-KIAN, with our curated policy fusion (Eq. (6)), obviates the need for manual temperature tuning for each task.

We compared TC-KIAN with the suggested variant: KIAN with the fusion temperature in Eq. (1) manually tuned (KIAN-T) (Figure 11, page 21). This will answer whether our trade loss and curated policy fusion, which automatically tunes the fusion temperature in Eq. (6), achieve better performance and reduce the need for manual temperature tuning.

Our experimental setup:

  • The default temperature parameter is 1 (KIAN), and we added the results for temperature = 0.5 (KIAN-T0.5) and 0.1 (KIAN-T0.1).
  • We evaluate KIAN, KIAN-T, TC-KIAN, and TC-KIAN w/o trade loss (C-KIAN) on four tasks: Doorkey-5x5, Dynamics-Obstacle-Random-6x6, MultiRoom-N4S5, KeyCorridor-S3R3.
  • We use KS-all as the external knowledge set, which is not preselected.

In Figure 11, we can observe:

  • In general, TC-KIAN outperforms KIAN-T, except in Dynamics-Obstacle-Random-6x6, where temperature 0.1 converges faster, but the final performance is close to TC-KIAN. This suggests that our trade loss and curated policy fusion, which automatically tunes the fusion temperature, improve performance.
  • No single temperature parameter is universally better than others. For example, temperature = 0.1 performs better in Dynamics-Obstacle-Random-6x6 but completely fails in KeyCorridor-S3R3.
  • KIAN-T can lead to reward curves with higher variance than TC-KIAN. This is especially obvious in Doorkey-5x5 and MultiRoom-N4S5.
  • KIAN-T can outperform C-KIAN in some tasks, implying that KIAN with careful temperature tuning can outperform KIAN with automatically tuned temperature. However, KIAN-T requires manual, task-specific tuning. In addition, TC-KIAN outperforms both methods, showing that our trade loss improves performance while requiring no manual temperature tuning.

We highlight that one of our main contributions is identifying the issue with a non-tuned fusion temperature in KIAN and proposing an automatic tuning method (our curated policy fusion) to address it. The original KIAN paper does not discuss that fusion temperature can affect performance, nor does it tune that parameter in its implementation. For a fair comparison, we follow their implementation closely and report KIAN results with temperature = 1.

Figures 7 and 11 show that adding our trade loss benefits performance, and this is not the only benefit. By learning correct relationships between knowledge representations, we can obtain more interpretable representations, as shown in Figures 6 and 8.

AuthorsOpenReview ↗

[Part 3] Author Response: Stress Tests on Structurally Misleading Knowledge

“Can the method handle systematically misleading external policies? Please include policies that output incorrect actions in some states, especially policies that fail at critical decision points, and test different interference ratios.”

Thank you for the suggestion! We ran the following experiments to stress-test TC-KIAN with knowledge policies that are misleading at different levels (Figure 12, page 21).

Misleading Knowledge Policy Design: For each knowledge policy $\pi_{g_j}$ in Appendix D, we design misleading versions of it at different levels, $\pi_{g_j}^{M,\ell}$, where $\ell \in \{1, 2, D\}$ represents the level of the misleading policies.
The following definitions hold for all knowledge policies in Appendix D except KG4 and KG5 (do_not_hit and do_not_hit_balls):

  • Level 1 ($\ell$ = 1): Follow $\pi_{g_j}$ until there is only one step away from an object, then choose any action except the one that $\pi_{g_j}$ suggests.
  • Level 2 ($\ell$ = 2): Follow $\pi_{g_j}$ until there are only two steps away from an object, then choose any direction to go except the one that $\pi_{g_j}$ suggests.
  • Level D ($\ell$ = D): Follow $\pi_{g_j}$ until the agent sees an object, then choose any direction to go except the one that $\pi_{g_j}$ suggests. (Here, D stands for the maximum distance at which the agent can see an object.)

For KG4 and KG5 (do_not_hit and do_not_hit_balls), their misleading versions are designed as follows:

  • If an agent sees an obstacle, it goes directly to the obstacle with probability 0.2 (Level 1), 0.4 (Level 2), or 1 (Level D).

For these knowledge policies, Level D is the most misleading.

Our experimental setup:

  • Our knowledge set is composed of all policies in KS-preselected + their misleading versions at Level 1, 2, or D.
  • We evaluate PPO, KIAN (KS-preselected), KIAN (KS-preselected + misleading), and TC-KIAN (KS-preselected + misleading) on two tasks: DoorKey-5x5 and MultiRoom-N4S5.

In Figure 12, we can observe:

  • TC-KIAN with misleading knowledge at all levels outperforms PPO in both environments, while KIAN performs worse than PPO under all misleading levels in DoorKey-5x5.
  • TC-KIAN’s performance degrades much less than KIAN’s. Specifically, TC-KIAN with and without misleading knowledge have comparable final performance. In contrast, KIAN with misleading knowledge converges more slowly, and its final performance is worse than KIAN without it.

These results indicate that TC-KIAN is robust to structurally misleading information and effectively distinguishes useful knowledge from misleading knowledge without careful preselection.

Reviewer_osXSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies knowledge-grounded reinforcement learning without expert knowledge preselection, focusing on the effects of irrelevant, duplicated, and noisy knowledge policies. Building on KIAN, the authors propose TC-KIAN. A trade loss aligns the geometry of knowledge keys with differences in policy behavior, while curated policy fusion uses query and key norms to separately control fusion weights and external policies' action distributions. Experiments on 12 MiniGrid tasks compare unselected and expert-selected knowledge settings, supplemented by stress tests with 100 knowledge policies, component ablations, representation analyses, and inference-time pruning. The results show improved learning under the evaluated unselected knowledge settings and reduced degradation from duplicated and uniform policies.

Critical strengths

The problem is practically relevant, and the two components have clear complementary roles. Removing the need to manually select knowledge for each task is a useful step toward more practical knowledge-grounded reinforcement learning. The trade loss provides behavioral relationship signals for knowledge keys, while curated fusion improves the discrimination of policy weights in the current state. The resulting design is coherent and integrates with the existing KIAN framework.

The experiments examine both performance and method behavior in useful detail. The paper evaluates 12 MiniGrid tasks with five random seeds and confidence intervals, and includes stress tests with duplicated and uniform policies, component ablations, and analyses of fusion weights and representation geometry. Runtime and inference-time pruning experiments provide additional evidence of practical utility, extending the evaluation beyond reward curves alone.

Critical weaknesses

The experimental validation of the central motivation could further separate relational learning, temperature adjustment, and auxiliary constraints. The existing component-removal ablations support the complementary roles of the trade loss and curated fusion. However, they do not fully establish how much of the benefit depends on correct behavioral relationships between policies, rather than a better attention temperature or a generic auxiliary constraint. On representative tasks, I suggest adding a KIAN control that calibrates only the attention temperature while retaining the original policy-fusion structure, and a trade-loss control that shuffles the correspondence between policy pairs and their behavioral differences while maintaining a comparable loss scale. Using the same tuning budget would help directly test the motivation that information trading creates useful knowledge structure.

The current stress tests cover a limited range of misleading knowledge. The base set contains eight handcrafted programs, and expansion to 100 policies mainly duplicates these programs or adds uniform action policies. These settings usefully test redundancy and uninformative policies, but provide limited evidence for behaviorally diverse knowledge that is systematically wrong or fails only in particular states. Within the existing MiniGrid tasks, I suggest introducing policies that are helpful in some states but confidently recommend incorrect actions in others, varying their proportion, and comparing reward learning curves and gains over knowledge-free PPO. This would more directly test robustness to structured misleading information and clarify the scope of the preselection-free claim.

The evaluation on only MiniGrid environment is not enough.

Questions for authors

Under the same tuning budget, how much of the performance and training-variability gap remains after calibrating only KIAN's attention temperature?

If the auxiliary loss scale is kept comparable but the correspondence between policy pairs and their behavioral differences is shuffled, how do representation structure and task performance change?

When knowledge is useful in some states but confidently incorrect in others, does TC-KIAN retain gains over knowledge-free PPO and avoid substantial negative transfer?

3 comments
AuthorsOpenReview ↗

Author Response - Overview

We thank the reviewer for their insightful comments! Please see our response below. We uploaded a revised manuscript that includes the additional experimental results (starting on page 21).

Information Trading v.s. Temperature Tuning
Please see [Part 1] below for the detailed response and additional experimental results.

Stress Tests on Structurally Misleading Knowledge
Please see [Part 2] below for the detailed response and additional experimental results.

Evaluation on Additional Environments

“The evaluation on only MiniGrid environment is not enough.”

We ran additional experiments on two tasks in ViZDoom, DefendLine and HealthGathering (Figure 15, page 22). The results show that TC-KIAN is more sample efficient when incorporated with a non-preselected knowledge set. TC-KIAN also reduces the performance gap relative to KIAN (KS-preselected), which is presumed to be our performance upper bound.

AuthorsOpenReview ↗

[Part 1] Author Response: Information Trading v.s. Temperature Tuning

“The existing component-removal ablations support the complementary roles of the trade loss and curated fusion. However, they do not fully establish how much of the benefit depends on correct behavioral relationships between policies, rather than a better attention temperature or a generic auxiliary constraint.”

Actually, in Figure 7, the results for TC-KIAN without trade loss (C-KIAN) show how much the performance improvement depends on knowledge representation learning, since trade loss is the main driver of learning these representations. Without trade loss, the agents perform worse in two challenging environments, MultiRoom-N4-S5 and KeyCorridorS3R3, showing that learning functionality-aligned knowledge representations improves performance.

In addition, our curated policy fusion (Eq. (6)), or C-KIAN, can be seen as automatically tuning the fusion temperature in KIAN (Eq. (1)), which obviates the need for manual temperature tuning for each task.

We compared TC-KIAN with the two suggested variants below.

“On representative tasks, I suggest adding a KIAN control that calibrates only the attention temperature while retaining the original policy-fusion structure, …”

Thank you for the suggestion! We ran the experiments with KIAN’s temperature manually tuned (KIAN-T). We have uploaded a revised manuscript to include this result (Figure 11, page 21).

Our experimental setup:

  • The default temperature parameter is 1 (KIAN), and we added the results for temperature = 0.5 (KIAN-T0.5) and 0.1 (KIAN-T0.1).
  • We evaluate KIAN, KIAN-T, TC-KIAN, and TC-KIAN w/o trade loss (C-KIAN) on four tasks: Doorkey-5x5, Dynamics-Obstacle-Random-6x6, MultiRoom-N4S5, KeyCorridor-S3R3.
  • We use KS-all as the external knowledge set, which is not preselected.

In Figure 11, we can observe:

  • In general, TC-KIAN outperforms KIAN-T, except in Dynamics-Obstacle-Random-6x6, where temperature 0.1 converges faster, but the final performance is close to TC-KIAN. This suggests that our trade loss and curated policy fusion, which automatically tunes the fusion temperature, improve performance.
  • No single temperature parameter is universally better than others. For example, temperature = 0.1 performs better in Dynamics-Obstacle-Random-6x6 but completely fails in KeyCorridor-S3R3.
  • KIAN-T can lead to reward curves with higher variance than TC-KIAN. This is especially obvious in Doorkey-5x5 and MultiRoom-N4S5.
  • KIAN-T can outperform C-KIAN in some tasks, implying that KIAN with careful temperature tuning can outperform KIAN with automatically tuned temperature. However, KIAN-T requires manual, task-specific tuning. In addition, TC-KIAN outperforms both methods, showing that our trade loss improves performance while requiring no manual temperature tuning.

We highlight that one of our main contributions is identifying the issue with a non-tuned fusion temperature in KIAN and proposing an automatic tuning method (our curated policy fusion) to address it. The original KIAN paper does not discuss that fusion temperature can affect performance, nor does it tune that parameter in its implementation. For a fair comparison, we follow their implementation closely and report KIAN results with temperature = 1.

Figures 7 and 11 show that adding our trade loss benefits performance, and this is not the only benefit. By learning correct relationships between knowledge representations, we can obtain more interpretable representations, as shown in Figures 6 and 8.

“… and a trade-loss control that shuffles the correspondence between policy pairs and their behavioral differences while maintaining a comparable loss scale.”
“..., how do representation structure and task performance change?”

Thank you for the suggestion! We ran the experiments with TC-KIAN’s policy-representation correspondence shuffled on LavaCrossingS9N2 (Figure 14, page 22). The external knowledge sets include KS-all and Extra-stresstest-U.

The results show that breaking the policy-representation correspondence worsens the performance when using both knowledge sets. In addition, as the number of knowledge policies grows, the performance gap widens. These results suggest that maintaining correct relationships between knowledge keys is a driving force of performance.

The tSNE plot in Figure 14 further suggests that breaking the policy-representation correspondence leads to no separation among representations of distinct knowledge, which hurts their interpretability.

AuthorsOpenReview ↗

[Part 2] Author Response: Stress Tests on Structurally Misleading Knowledge

“The current stress tests cover a limited range of misleading knowledge. …, I suggest introducing policies that are helpful in some states but confidently recommend incorrect actions in others, varying their proportion, and comparing reward learning curves and gains over knowledge-free PPO…”

Thank you for the suggestion! We ran the following experiments to stress-test TC-KIAN with knowledge policies that are misleading at different levels (Figure 12, page 21).

Misleading Knowledge Policy Design: For each knowledge policy $\pi_{g_j}$ in Appendix D, we design misleading versions of it at different levels, $\pi_{g_j}^{M,\ell}$, where $\ell \in \{1, 2, D\}$ represents the level of the misleading policies.
The following definitions hold for all knowledge policies in Appendix D except KG4 and KG5 (do_not_hit and do_not_hit_balls):

  • Level 1 ($\ell$ = 1): Follow $\pi_{g_j}$ until there is only one step away from an object, then choose any action except the one that $\pi_{g_j}$ suggests.
  • Level 2 ($\ell$ = 2): Follow $\pi_{g_j}$ until there are only two steps away from an object, then choose any direction to go except the one that $\pi_{g_j}$ suggests.
  • Level D ($\ell$ = D): Follow $\pi_{g_j}$ until the agent sees an object, then choose any direction to go except the one that $\pi_{g_j}$ suggests. (Here, D stands for the maximum distance at which the agent can see an object.)

For KG4 and KG5 (do_not_hit and do_not_hit_balls), their misleading versions are designed as follows:

  • If an agent sees an obstacle, it goes directly to the obstacle with probability 0.2 (Level 1), 0.4 (Level 2), or 1 (Level D).

For these knowledge policies, Level D is the most misleading.

Our experimental setup:

  • Our knowledge set is composed of all policies in KS-preselected + their misleading versions at Level 1, 2, or D.
  • We evaluate PPO, KIAN (KS-preselected), KIAN (KS-preselected + misleading), and TC-KIAN (KS-preselected + misleading) on two tasks: DoorKey-5x5 and MultiRoom-N4S5.

In Figure 12, we can observe:

  • TC-KIAN with misleading knowledge at all levels outperforms PPO in both environments, while KIAN performs worse than PPO under all misleading levels in DoorKey-5x5.
  • TC-KIAN’s performance degrades much less than KIAN’s. Specifically, TC-KIAN with and without misleading knowledge have comparable final performance. In contrast, KIAN with misleading knowledge converges more slowly, and its final performance is worse than KIAN without it.

These results indicate that TC-KIAN is robust to structurally misleading information and effectively distinguishes useful knowledge from misleading knowledge without careful preselection.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To AC and All Reviewers

We thank the reviewers for their thoughtful feedback. We are glad that they found the problem we study, removing the need for per-task manual knowledge selection in Knowledge-Grounded RL, to be practically meaningful (all reviewers), and that the motivation is clear and sound (Reviewer RRkL). We are encouraged that they found our method to have a clear design rationale and be conceptually sound (all reviewers), and that our design is coherent and integrates well with existing frameworks (Reviewers osXS and RRkL). We are pleased that the reviewers recognize the breadth and multifacetedness of our experiments (all reviewers), including reward curves, stress tests with duplicated and random knowledge, component ablations, attention weights and representation visualization, computational analysis, and inference-time pruning.

The primary concerns are

  • the lack of performance-gain analysis on merely tuning KIAN’s attention temperature,
  • the lack of robustness stress tests with structurally misleading knowledge policies, and
  • an insufficient connection between the theory and the practical algorithm.

We address the reviewers’ comments below and have added the results to the revised manuscript (starting on page 21).

To summarize,

  • (Tuning KIAN’s Attention Temperature) We clarify that TC-KIAN without trade loss in Figure 7 can be seen as automatically tuning KIAN’s attention temperature. We provide additional experiments (Figure 11, page 21) showing that TC-KIAN's trade loss and curated policy fusion, which automatically tunes the fusion temperature, achieve better performance and reduce the need for manual temperature tuning.
  • (Stress Tests with Structurally Misleading Knowledge Policies) We provide additional experiments (Figure 12, page 21) showing that TC-KIAN is more robust to misleading information and effectively distinguishes useful knowledge from misleading knowledge without careful preselection.
  • (Connection between The Theory and The Practical Algorithm) We explain how the bounded key norm assumption can be realized in our algorithm. We also provide additional experiments (Figure 13, page 21) showing that TC-KIAN with bounded key norms does not affect convergence and, in general, performs better than TC-KIAN without bounded key norms.

Dear Reviewers,

Thank you for your reviews. The authors have submitted their rebuttal. Could you please take a look and check whether it addresses your questions and concerns? Please leave a comment for the authors, and update your review if your assessment has changed.

Thank you again for your time and effort!

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.