- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This manuscript addresses the critical limitation in KGRL regarding the reliance on human preselection of task-relevant knowledge policies, and proposes a preselection-free method named TC-KIAN based on the KIAN framework. The proposed method introduces an auxiliary loss based on the pairwise KL divergence between policies to regularize knowledge representations, alongside a policy fusion mechanism that utilizes query and key norms to modulate fusion weights and distribution temperatures. The manuscript achieves superior performance over baselines including KIAN across 12 MiniGrid tasks as well as stress tests with redundant and uniform knowledge policies. I find that the manuscript possesses certain research value, but it still requires essential theoretical clarifications and controlled ablation experiments to substantiate its core claims.
Critical strengths
- The manuscript identifies an important and practically meaningful research problem by focusing on the limitation of existing KGRL methods that rely on human-preselected knowledge, shifting the research focus toward the effective fusion of unselected knowledge sets containing redundancy and distraction. The motivation is clear and sound.
- Explicitly incorporating behavioral similarity between policies into knowledge representation learning is conceptually sound. This design mitigates the complete reliance of knowledge representation learning on sparse environmental rewards to a certain extent.
- The engineering motivation behind Curated Policy Fusion is intuitive, as it identifies the limitation of the normalized inner product in the original KIAN where attention weights become diluted as the knowledge repository grows. Furthermore, it demonstrates good compatibility with existing frameworks.
- The experimental evaluation not only includes primary performance comparisons, but also designs stress tests with duplicated and uniform policies, ablation studies on the two core modules, and visualizations of attention weights and representations. These experiments provide a multifaceted empirical analysis.
Critical weaknesses
- The core auxiliary loss, Trade Loss, may be mathematically unbounded from below, and the theoretical assumptions in the appendix diverge significantly from the actual algorithm. In Equations (3) to (5), without explicit constraints on the norms of knowledge keys or pairwise distances, when there exist policy pairs whose KL divergence exceeds the mean threshold , the repulsive term can drive by infinitely scaling up the norms of the keys. In addition, Appendix A.1 assumes that the attractive and repulsive terms share a common finite optimal solution, which rarely holds in an unconstrained space. In my view, for an objective function that could be unbounded from below, the continuous decrease of the loss observed in Figure 5c does not naturally equate to a stable improvement in representation quality.
- The empirical attribution of the mechanism is not entirely clean, as the gains from simple attention temperature adjustments have not been fully ruled out. The original KIAN employs an inner product of unit vectors where logits are constrained to , which limits the theoretical maximum attention weight of a single useful policy to only about under the stress test with 100 distracting policies. TC-KIAN preserves the query norm, which essentially introduces a learnable inverse temperature parameter, while Equation (7) also utilizes the key norm to adjust the temperature of the action distributions of knowledge policies. Although Figure 7 provides module ablations, critical and carefully controlled baselines are missing. In my opinion, the current experiments do not sufficiently prove whether the performance improvement stems from sophisticated knowledge representation learning or merely from overcoming the restricted temperature limitation of the original KIAN.
- I find that the claim of being preselection-free is overstated, because the current experiments hardly support the generalizability of this conclusion under realistic and complex knowledge repositories. The existing evaluations are conducted exclusively on eight handcrafted, isomorphic rule based policies designed for MiniGrid. The stress tests merely perform exact duplicate sampling of these eight policies or inject uninformative uniform random policies. This setup is far removed from real world scenarios that feature contaminated knowledge with high-confidence errors, partially detrimental states, or conflicting policy objectives.
- The arguments regarding computational scalability and sparse selection are not sufficiently rigorous. Trade Loss requires computing the pairwise KL divergence across all policy pairs, which incurs a quadratic complexity of with respect to the number of knowledge policies . Meanwhile, Equations (6) and (7) still perform dense forward passes and weighted summation across all knowledge policies, and concentrated attention weights do not equate to achieving computational sparse selection or automated pruning. Furthermore, the pruning experiments in Appendix G rely on manually selecting representative policies from three categories, namely PK, OD, and G, which fails to demonstrate that the model can automatically identify the optimal subset during inference.
- The manuscript also exhibits weaknesses regarding statistical evidence and baseline configurations. Specifically, several learning curves in Figure 3 and Figure 7 exhibit high variance with noticeably overlapping confidence intervals. Moreover, the calculation methodology for the claimed 89.6% performance gain remains ambiguous. Crucial implementation details are also missing, such as whether the pairwise policy loss is summed or averaged across pairs.
Questions for authors
- I would like the authors to clearly explain how the unbounded growth of the key norm is prevented during actual training. Could you provide the empirical trajectory of the knowledge key norms during training, and explain how the theoretical assumptions in Appendix A can be revised to align with the practical algorithm?
- Could the authors provide a direct comparative experiment with KIAN plus a learnable or tuned temperature, as well as ablation results where the query norm and key norm are individually fixed, to demonstrate the independent contribution of the Trade Mechanism over simple temperature adjustment?
- Please provide numerical tables corresponding to the core experiments, along with the comparative results for TC-KIAN under an expert-selected knowledge set.
- Please clarify the normalization scheme for the summation over policy pairs in Equation (3), as well as the numerical smoothing details when computing the KL divergence for deterministic rule-based policies.
9 comments
[Part 1] Author Response: Connection between Theory and the Practical Algorithm
“The core auxiliary loss, Trade Loss, may be mathematically unbounded from below, and the theoretical assumptions in the appendix diverge significantly from the actual algorithm.”
“Could you provide the empirical trajectory of the knowledge key norms during training, and explain how the theoretical assumptions in Appendix A can be revised to align with the practical algorithm?”
Thank you for pointing this out. First, we clarify that only the bounded key norm assumption affects whether the trade loss is bounded below. Other assumptions are added mainly to study non-trivial cases (e.g., minimum separation = 0 implies trade loss = 0) and to perform theoretical analysis on a locally smooth function (Lemmas 2 and 3). This assumption has limited effect on whether the trade loss is bounded. From now on, we will focus on the bounded key norm assumption.
In our experiments, the bounded key norm assumption is never violated (Figure 5c), so whether the key norms are bounded has limited effect on the convergence performance of our current results. We did not include this constraint in our implementation because we want to follow KIAN as closely as possible to disentangle the constraint's effects from others. However, the bounded key norm assumption is accessible and easy to satisfy; for example, we can pass our learnable key vectors through $\alpha \cdot tanh(k_j)$. In this case, the key norm is bounded by $\alpha$.
We evaluated TC-KIAN and TC-KIAN (bounded keys, = 10) on Doorkey-5x5 (Figure 13, page 21). The external knowledge sets include KS-all, Extra-stresstest-R, and Extra-stresstest-U. The results show that TC-KIAN with bounded key norms does not affect convergence and, in general, performs better than TC-KIAN without bounded key norms.
“..., the continuous decrease of the loss observed in Figure 5c does not naturally equate to a stable improvement in representation quality.”
We clarify that Figure 5c is NOT meant to show improvement in representation quality. Figures 6 and 8 are more related to that. Figure 5c shows two things:
- Simultaneously minimizing the trade loss and maximizing the rewards does not lead to unstable knowledge key training. This implies that the two objectives are not contradictory, which is important for stable learning.
- KIAN also implicitly minimizes the trade loss while seeking better performance. This suggests that explicitly minimizing the same loss in TC-KIAN may explain our performance improvement.
[Part 2] Author Response: Comparison with Temperature Tuning
“..., critical and carefully controlled baselines are missing. In my opinion, the current experiments do not sufficiently prove whether the performance improvement stems from sophisticated knowledge representation learning or merely from overcoming the restricted temperature limitation of the original KIAN.”
Actually, in Figure 7, the results for TC-KIAN without trade loss (C-KIAN) show how much the performance improvement depends on knowledge representation learning, since trade loss is the main driver of learning these representations. Without trade loss, the agents perform worse in two challenging environments, MultiRoom-N4-S5 and KeyCorridorS3R3, showing that learning functionality-aligned knowledge representations improves performance.
In addition, our curated policy fusion (Eq. (6)), or C-KIAN, can be seen as automatically tuning the fusion temperature in KIAN (Eq. (1)), which overcomes the restricted temperature limitation in KIAN while obviating the need for manual temperature tuning for each task.
“Could the authors provide a direct comparative experiment with KIAN plus a learnable or tuned temperature, as well as ablation results where the query norm and key norm are individually fixed, to demonstrate the independent contribution of the Trade Mechanism over simple temperature adjustment?”
TC-KIAN w/o trade loss (C-KIAN) in Figure 7 can be seen as KIAN plus a learnable temperature. By directly comparing Eq. (1) and Eq. (6), the only difference is that Eq. (6) has $\| \mathbf{u}_t \|$ in the softmax, which is a learnable value.
We note that KIAN with fixed query and key norms causes entropy imbalance among knowledge policies, leading the agent to always choose the inner policy and ignore the external ones. This issue is detailed and discussed in the original KIAN paper.
We compared TC-KIAN with the suggested variant: KIAN with the fusion temperature in Eq. (1) manually tuned (KIAN-T) (Figure 11, page 21). This will answer whether our trade loss and curated policy fusion, which automatically tunes the fusion temperature in Eq. (6), achieve better performance and reduce the need for manual temperature tuning.
Our experimental setup:
- The default temperature parameter is 1 (KIAN), and we added the results for temperature = 0.5 (KIAN-T0.5) and 0.1 (KIAN-T0.1).
- We evaluate KIAN, KIAN-T, TC-KIAN, and TC-KIAN w/o trade loss (C-KIAN) on four tasks: Doorkey-5x5, Dynamics-Obstacle-Random-6x6, MultiRoom-N4S5, KeyCorridor-S3R3.
- We use KS-all as the external knowledge set, which is not preselected.
In Figure 11, we can observe:
- In general, TC-KIAN outperforms KIAN-T, except in Dynamics-Obstacle-Random-6x6, where temperature 0.1 converges faster, but the final performance is close to TC-KIAN. This suggests that our trade loss and curated policy fusion, which automatically tunes the fusion temperature, improve performance.
- No single temperature parameter is universally better than others. For example, temperature = 0.1 performs better in Dynamics-Obstacle-Random-6x6 but completely fails in KeyCorridor-S3R3.
- KIAN-T can lead to reward curves with higher variance than TC-KIAN. This is especially obvious in Doorkey-5x5 and MultiRoom-N4S5.
- KIAN-T can outperform C-KIAN in some tasks, implying that KIAN with careful temperature tuning can outperform KIAN with automatically tuned temperature. However, KIAN-T requires manual, task-specific tuning. In addition, TC-KIAN outperforms both methods, showing that our trade loss improves performance while requiring no manual temperature tuning.
We highlight that one of our main contributions is identifying the issue with a non-tuned fusion temperature in KIAN and proposing an automatic tuning method (our curated policy fusion) to address it. The original KIAN paper does not discuss that fusion temperature can affect performance, nor does it tune that parameter in its implementation. For a fair comparison, we follow their implementation closely and report KIAN results with temperature = 1.
Figures 7 and 11 show that adding our trade loss benefits performance, and this is not the only benefit. By learning correct relationships between knowledge representations, we can obtain more interpretable representations, as shown in Figures 6 and 8.
Author Response - Overview
We thank the reviewer for their insightful comments! Please see our response below. We uploaded a revised manuscript that includes the additional experimental results (starting on page 21).
Connection between Theory and the Practical Algorithm
Please see [Part 1] below for the detailed response and additional experimental results.
Comparison with Temperature Tuning
Please see [Part 2] below for the detailed response and additional experimental results.
Stress Tests on Structurally Misleading Knowledge
Please see [Part 3] below for the detailed response and additional experimental results.
Computational Scalability and Sparse Selection
We DO NOT claim that our method is performing sparse selection or automated pruning. Your understanding of the forward passes is correct, but we never claimed in our paper that our attention weights on knowledge policies are related to sparse selection or automated pruning.
Our computational analysis in Appendix F shows that, even with the additional trade-loss computation, the computational overhead is minimal. This is unrelated to sparse selection.
Our knowledge pruning analysis in Appendix G is an ablation study that validates whether the key representations learned by the trade loss enable robust performance under knowledge pruning during inference. We adopt an autonomous, not manual, pruning method derived from prior work in explainable ML [1]. Our purpose is to analyze the effects of our trade loss, and we DO NOT claim to propose any autonomous knowledge pruning method in this paper.
We believe this comment reflects a major misunderstanding of our work, and we kindly ask the reviewer to reread the paper for clarification.
[1] Yi-Lin Tuan, Connor Pryor, Wenhu Chen, Lise Getoor, and William Yang Wang. Local explanation of dialogue response generation. Advances in Neural Information Processing Systems, 34:404–416, 2021.
Statistical Evidence, Metric for Improvement Percentages, and Implementation Details
Thank you for pointing this out. We discussed the learning variance in our original limitations section (Appendix H): “TC-KIAN’s learning-curve variance is sometimes higher than that of KIAN, the state-of-the-art KGRL method, showing that while it converges faster and achieves higher rewards, its stability can be improved in future work.”
We also note that most of our results are considered statistically significant improvements since only a small portion of confidence intervals overlap.
For Figure 3, we provide the table of average rewards and improvement percentages in the revised manuscript (Table 3, page 22).
In Figure 4 of the original manuscript, we reported the results of TC-KIAN with an expert-selected knowledge set (KS-preselected).
Thank you for pointing this out! We took the average after summing the pairwise trade loss across policy pairs. We will add this implementation detail. Note that whether to take the average in Eq. (3) has limited effect on the learning results, as it merely adds a constant to the function.
We clarify that our knowledge policies are all stochastic. Each of them is a probability vector with each component >= 0 and <= 1, and the sum of all components = 1. Therefore, we can naturally compute KL divergence on these probabilities.