- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies reinforcement learning with incomplete symbolic knowledge in parametrized action spaces and proposes KGRL. A Datalog knowledge base derives state-dependent restrictions on discrete actions and bounds on their continuous parameters. PARAMOPT then refines the parameters using gradients of the learned Q-function. Knowledge constrains both action selection and TD targets, while symbolic derivations provide rule-based traces. The evaluation covers four environment families, five parametrized-action RL baselines, and ablations of knowledge guidance, gradient guidance, and their application at different stages.
Critical strengths
-
A meaningful research problem. Complete symbolic environment models are often unavailable, while partial expert knowledge can still eliminate unreasonable decisions. Integrating such knowledge into RL with both discrete action choices and continuous parameters is a worthwhile direction.
-
Broad experimental coverage. Multiple environments, five external baselines, eight random seeds, and multiple ablations help analyze the roles of knowledge and gradient guidance at different stages.
Critical weaknesses
-
The incremental gains from GG are very small, leaving the added value of the combined design insufficiently established. In Table 1, removing GG during both training and evaluation produces results close to the complete method, with better mean performance on some tasks. The authors also acknowledge task-dependent benefits. The evidence therefore supports knowledge constraints more strongly than the necessity of gradient-based parameter refinement. This limits the demonstrated additional contribution of the combined design. The construction effort and cross-task reuse of the handcrafted rules and state abstractions are also not quantified.
-
Baseline comparisons involve unequal access to knowledge and implementation correctness concerns.
Unequal knowledge access: Most of KGRL's gains may come from the additional expert knowledge, while its algorithmic design may contribute relatively little. The external baselines do not receive equivalent knowledge, making the comparison unfair in this respect. I suggest adding some baselines that can also use external expert knowledge, such as P-DQN/MP-DQN augmented with expert knowledge, or other knowledge-guided RL/NeSy methods, including suitably adapted methods originally designed for non-parametrized action spaces. This would help establish whether KGRL provides additional gains.
Implementation correctness: In the supplementary P-DQN and MP-DQN implementations, replay does not retain the executed continuous parameters. Critic updates instead recompute parameters using the current actor. The action supplied to the Q-function can therefore differ from the complete action that generated the sampled reward and transition. Relevant locations are lines 389–396 and 439–443 of the P-DQN training implementation, and lines 395–402 and 445–449 for MP-DQN. This raises concerns about baseline reliability, although I cannot determine whether the reported experiments used this version or quantify its impact.
Questions for authors
-
Comparison with baselines using expert knowledge. Could the authors add representative baselines that can also use external expert knowledge, such as P-DQN/MP-DQN augmented with the same knowledge, or other knowledge-guided RL/NeSy methods, including methods originally designed for non-parametrized action spaces with suitable action-space adaptations? A representative subset would suffice; adapting every baseline is unnecessary. For knowledge-augmented P-DQN/MP-DQN, knowledge could likewise constrain action selection, continuous parameters, exploration, and TD targets under comparable budgets to test whether KGRL retains a consistent advantage. Other methods should also receive the same or equivalent knowledge.
-
Clarification and correction of the baseline code in the supplementary material. The P-DQN and MP-DQN implementations in the supplementary code do not store the executed continuous parameters in replay and instead recompute them using the current actor during critic updates. Is this intentional? Were the experiments reported in the paper conducted using these implementations from the supplementary material? Please explain the justification or, if erroneous, provide corrected comparisons.
-
Evaluation in more complex environments. The current evaluation environments are relatively simple. Could the authors add an experiment on a more challenging environment with parametrized actions, for example one with richer visual observations, longer episodes, or larger state and action spaces, to assess whether KGRL's performance advantages persist?
2 comments
Thank you for correcting the baseline code, comparing methods with the same expert knowledge, and testing on harder tasks. GG’s added benefit remains unclear, but I’ve decided to raise my score.
We thank the reviewer for the careful review. Most importantly, we corrected the implementations and added comparisons under identical knowledge, alternative estimators. Therefore, we added substantial experiments with three new evaluation domains that describe navigation task with obstacles (Bottleneck, ZigZag, and UNav) to the Appendix of the updated paper. These experiments are only performed on 3 seeds and 300k trainin steps. A full sweep across 8 seeds and 500k training steps is currently running and we will update the results in the main paper and appendix accordingly, as soon as it is finished.
Weakness 1. Contribution of gradient-guidance and handcrafted knowledge “The incremental gains from GG are very small.” “Most of KGRL's gains may come from the additional expert knowledge.”
We added P-DQN-KB and MP-DQN-KB with identical knowledge (Appendix B.2, Table 5, Figure 5). On UNav, KGRL first reaches 100% success at 20.0±0.0k interactions, versus P-DQN-KB's 73.3±49.3k and MP-DQN-KB's 90.0±10.0k. Its mean AULC is 9.663±0.119 versus 8.845±0.642 and 8.167±0.389, respectively. On ZigZag, KGRL and MP-DQN-KB reach 100% final success with comparable AULC, while P-DQN-KB reaches 65.6±42.0%. MP-DQN-KB is stronger on Bottleneck, which shows that the advantages are domain-dependent.
Weakness 2. Baseline implementation correctness “Replay does not retain the executed continuous parameters.”
We confirmed and corrected this error in our local adaptations of P-DQN and MP-DQN. The replay buffer now retains the parameter vector, and critic regression uses the action that generated each reward and transition. We also corrected MP-DQN's action-specific multi-pass outputs. Appendix B.1, Table 3 reports fresh comparisons using the corrected implementations under a shared 300k-interaction budget. KGRL achieves higher mean return AULC than both corrected baselines on eight of nine tasks. Platform is the exception, where corrected MP-DQN is stronger.
Question 1. Comparisons with knowledge-augmented baselines?
P-DQN-KB and MP-DQN-KB apply the same knowledge guidance during exploration, deployment. Table 5 and Figure 5 in the appendix report the results. Our answer to Weakness 1 reports KGRL's remaining advantage.
Question 2. Was recomputing replay parameters intentional, and which code produced the original results?
It was an unintentional error in our local P-DQN and MP-DQN adaptations. We corrected the error as described in our answer to Weakness 2. The original evaluation logs do not preserve the training-source version, so those records cannot establish which implementation produced the original results. The new results in Table 3 use the corrected code and independently support the comparison.
Question 3. Evaluation in more complex environments?
We introduced three additional diagnostic domains (Bottleneck, ZigZag, and UNav), described in Appendix A.1. The required trajectories to solve the domains are substantially longer than the ones of the benchmark domains. ZigZag requires successive detours around alternating barriers, UNav requires temporarily moving away from the goal to enter the obstacle's opening. At 300k interactions, KGRL achieves 95.8±4.8%, 100±0%, and 100±0% final success. The vanilla version of MP-DQN reaches 0% on all three domains, and P-DQN reaches 1.0±1.8%, 0%, and 0% (Table 3). Figure 8 additionally shows 0/288 final collision episodes for KGRL versus 213/288 for MP-DQN and 112/288 for P-DQN. The equal-knowledge results in our answer to Weakness 2 separate the contribution of supplied guidance from KGRL's additional learning benefit.