- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces NeuSAT for learning and transferring neuro-symbolic state abstractions from visual observations. Stage 1 uses reinforcement learning to build a visual abstraction tree and its action values in a source environment. Stage 2 uses an LLM-generated pixel parser and proposition vocabulary to build a proposition-based abstraction tree under the guidance of the Stage 1 teacher. Stage 3 transfers this tree and its action values to a target environment, updates the values through interaction, and adds local visual branches when needed. Experiments on MiniGrid-DoorKey, Crafter, and Sokoban evaluate source learning, target transfer, and the roles of the different stages.
Critical strengths
-
The method is appealing and offers some novelty. The paper examines how an abstraction suited to source learning can be turned into one suited to transfer across environments. It connects visual abstraction learning, teacher-guided semantic distillation, and local refinement in the target environment into a complete process. This design has a clear motivation and offers a valuable research direction for neuro-symbolic reinforcement learning.
-
The reported performance is strong, and the experimental coverage is broad. The authors evaluate 15 unseen target environments and five random seeds in each of three domains, comparing against PPO from scratch, PPO fine-tuning, JSRL, and CKA-RL. The paper reports that NeuSAT reaches 90% success within the budget in every evaluated target run and achieves the highest area under the success-rate curve in all three domains. Ablations on direct visual transfer, value initialization, and target refinement also help examine the roles of the framework's components.
Critical weaknesses
-
The additional benefit of the Stage 1 teacher is unclear. The evaluation lacks a direct comparison that uses the same pixel parser and proposition vocabulary but trains a symbolic policy through reinforcement learning without Stage 1 teacher supervision. Without this control, it is difficult to determine how much additional benefit teacher guidance provides to Stage 2 and subsequent transfer.
-
The contribution of pixel-to-symbol conversion to the performance gains is unclear. Converting pixel observations into a symbolic representation may itself make the task easier to learn. The evaluation lacks other RL baselines using the same parser and symbolic inputs. It therefore remains unclear how much improvement comes from this conversion itself and how much additional benefit comes from NeuSAT's other learning and transfer mechanisms.
Questions for authors
-
How much additional benefit does the Stage 1 teacher provide? Could the authors keep the same pixel parser and proposition vocabulary, train a symbolic policy directly through reinforcement learning without Stage 1 teacher supervision, and then compare its target adaptation with full NeuSAT under comparable conditions? Please specify the policy representation and the source and target interaction budgets, and account for the additional sampling cost of Stage 2.
-
How do baselines with the same symbolic inputs perform? Could the authors add baselines using the same parser and proposition vocabulary, such as PPO operating on proposition vectors or a simple symbolic RL method, and compare their transfer performance? This would clarify how much additional benefit teacher guidance, tree distillation, and target refinement provide when the symbolic inputs are shared. One well-designed comparison could also help answer both questions.
4 comments
Clarifying the Contributions of Teacher Guidance and Semantic Representations
We thank the reviewer again for raising two important questions: how much Stage 1 teacher guidance contributes to transfer, and whether semantic representations alone explain NeuSAT's performance gains. Our new ablations attached above directly address both questions. Direct-TP demonstrates the benefit of teacher-guided distillation, while PPO(P)/DQN(P) show that identical semantic inputs alone are insufficient to achieve NeuSAT's transfer performance.
We would also like to further clarify the role of propositions in our abstraction-learning process. State abstraction learning requires both identifying how states can be grouped and determining which groupings are decision-relevant. Propositions provide a semantic bias for the former, while value information learned through interaction and Stage 1 teacher guidance determines the latter.
We have incorporated these experiments into Section 5.4 of the revised manuscript (highlighted in blue). We sincerely appreciate the reviewer's constructive feedback and welcome further discussion.
Thank you for the response, which addresses my concerns. I will update my score if necessary.
Thank You for Your Feedback and Continued Discussion
We sincerely thank the reviewer for their thoughtful feedback and for taking the time to consider our responses. With a few hours remaining in the discussion period, we would be very happy to clarify any further questions or concerns.
We would also greatly appreciate a score update if the reviewer feels it is appropriate. Thank you again for your constructive engagement.
NeuSAT Outperforms Direct Propositional Learning and PPO/DQN with Identical Propositional Inputs: New Ablation Results
We thank the reviewer for the positive assessment and for highlighting these important experimental gaps. We agree that isolating the contributions of teacher guidance and semantic representations is essential to validating NeuSAT. To address both concerns, we conducted two additional experiments and incorporated them into Section 5.4 of the revised manuscript (highlighted in blue). Together, the results show that LLM-generated propositions alone are insufficient to reproduce the transfer performance of the complete NeuSAT framework. Value-guided abstraction learning and teacher-guided distillation help construct more effective transferable abstractions.
Specifically, we compare NeuSAT with three alternatives: Direct-TP, which learns propositional splits without the Stage 1 teacher, and PPO(P)/DQN(P), which learn policies using exactly the same parser and proposition vocabulary as NeuSAT. Direct-TP matches NeuSAT's total source interaction budget and uses the same Stage 3 adaptation. All methods are evaluated on 15 held-out targets per domain, with 150k target interactions.
Table 1 presents our ablation study on teacher guidance and semantic representations. All three alternatives underperform NeuSAT, particularly on Crafter and Sokoban. The results address the reviewer's two questions:
(1) Benefit of the Stage 1 teacher: Direct-TP performs worse despite using the same propositions, tree representation, target adaptation, and matched source interaction budget. This supports the value of first learning decision-relevant distinctions through source value learning and then reconstructing them through teacher-guided distillation.
(2) Benefit beyond semantic inputs: PPO(P)/DQN(P) successfully solve the source tasks but transfer substantially less effectively. Accurate propositions do not necessarily capture all decision-relevant distinctions, and states sharing the same proposition vector may require different actions.
Table 1: Ablation Study on Teacher Guidance and Semantic Representations
*Metrics follow Table 1 of the paper. NeuSAT results use five evaluation seeds; Direct-TP uses five independent source seeds.