acceptodds
Under review as a conference paper at ICLR 2027

Learning Neuro-Symbolic Abstractions for Transfer in Reinforcement Learning

Abstract

Reinforcement learning agents often require substantial interaction to adapt to a new environment, even when the new environment preserves the task semantics of a previously solved source environment. Reusing knowledge acquired in the source can reduce this cost, while doing so requires the agent to identify states that play the same decision-relevant role across environments. State abstractions provide a natural way to share action-value estimates and policies across such states, while abstractions induced by value signals from the source environment may encode layout-specific distinctions and therefore transfer poorly. We introduce Neuro-symbolic Abstraction and Transfer (NeuSAT), a single-source online transfer framework built on the observation that an abstraction effective for source learning is not necessarily suitable for transfer. NeuSAT first learns a visually routed abstraction tree and an abstract policy through interaction with the source environment. It then uses them as a teacher to rebuild the state partition with auto-generated semantic propositions and fits abstract action values on the resulting proposition-routed tree. The propositional tree and its action values are transferred to unseen target environments as a zero-shot policy prior. During the agent's interaction with the target environment, NeuSAT updates the transferred values and locally refines leaves that combine states with conflicting action preferences. Our evaluation results on three domains across held-out layouts and environment scales show that proposition-structured state abstractions can support sample-efficient transfer from a single source environment.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, State Abstraction, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fXJUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces NeuSAT for learning and transferring neuro-symbolic state abstractions from visual observations. Stage 1 uses reinforcement learning to build a visual abstraction tree and its action values in a source environment. Stage 2 uses an LLM-generated pixel parser and proposition vocabulary to build a proposition-based abstraction tree under the guidance of the Stage 1 teacher. Stage 3 transfers this tree and its action values to a target environment, updates the values through interaction, and adds local visual branches when needed. Experiments on MiniGrid-DoorKey, Crafter, and Sokoban evaluate source learning, target transfer, and the roles of the different stages.

Critical strengths
  1. The method is appealing and offers some novelty. The paper examines how an abstraction suited to source learning can be turned into one suited to transfer across environments. It connects visual abstraction learning, teacher-guided semantic distillation, and local refinement in the target environment into a complete process. This design has a clear motivation and offers a valuable research direction for neuro-symbolic reinforcement learning.

  2. The reported performance is strong, and the experimental coverage is broad. The authors evaluate 15 unseen target environments and five random seeds in each of three domains, comparing against PPO from scratch, PPO fine-tuning, JSRL, and CKA-RL. The paper reports that NeuSAT reaches 90% success within the budget in every evaluated target run and achieves the highest area under the success-rate curve in all three domains. Ablations on direct visual transfer, value initialization, and target refinement also help examine the roles of the framework's components.

Critical weaknesses
  1. The additional benefit of the Stage 1 teacher is unclear. The evaluation lacks a direct comparison that uses the same pixel parser and proposition vocabulary but trains a symbolic policy through reinforcement learning without Stage 1 teacher supervision. Without this control, it is difficult to determine how much additional benefit teacher guidance provides to Stage 2 and subsequent transfer.

  2. The contribution of pixel-to-symbol conversion to the performance gains is unclear. Converting pixel observations into a symbolic representation may itself make the task easier to learn. The evaluation lacks other RL baselines using the same parser and symbolic inputs. It therefore remains unclear how much improvement comes from this conversion itself and how much additional benefit comes from NeuSAT's other learning and transfer mechanisms.

Questions for authors
  1. How much additional benefit does the Stage 1 teacher provide? Could the authors keep the same pixel parser and proposition vocabulary, train a symbolic policy directly through reinforcement learning without Stage 1 teacher supervision, and then compare its target adaptation with full NeuSAT under comparable conditions? Please specify the policy representation and the source and target interaction budgets, and account for the additional sampling cost of Stage 2.

  2. How do baselines with the same symbolic inputs perform? Could the authors add baselines using the same parser and proposition vocabulary, such as PPO operating on proposition vectors or a simple symbolic RL method, and compare their transfer performance? This would clarify how much additional benefit teacher guidance, tree distillation, and target refinement provide when the symbolic inputs are shared. One well-designed comparison could also help answer both questions.

4 comments
AuthorsOpenReview ↗

NeuSAT Outperforms Direct Propositional Learning and PPO/DQN with Identical Propositional Inputs: New Ablation Results

We thank the reviewer for the positive assessment and for highlighting these important experimental gaps. We agree that isolating the contributions of teacher guidance and semantic representations is essential to validating NeuSAT. To address both concerns, we conducted two additional experiments and incorporated them into Section 5.4 of the revised manuscript (highlighted in blue). Together, the results show that LLM-generated propositions alone are insufficient to reproduce the transfer performance of the complete NeuSAT framework. Value-guided abstraction learning and teacher-guided distillation help construct more effective transferable abstractions.

Specifically, we compare NeuSAT with three alternatives: Direct-TP, which learns propositional splits without the Stage 1 teacher, and PPO(P)/DQN(P), which learn policies using exactly the same parser and proposition vocabulary as NeuSAT. Direct-TP matches NeuSAT's total source interaction budget and uses the same Stage 3 adaptation. All methods are evaluated on 15 held-out targets per domain, with 150k target interactions.

Table 1 presents our ablation study on teacher guidance and semantic representations. All three alternatives underperform NeuSAT, particularly on Crafter and Sokoban. The results address the reviewer's two questions:

(1) Benefit of the Stage 1 teacher: Direct-TP performs worse despite using the same propositions, tree representation, target adaptation, and matched source interaction budget. This supports the value of first learning decision-relevant distinctions through source value learning and then reconstructing them through teacher-guided distillation.

(2) Benefit beyond semantic inputs: PPO(P)/DQN(P) successfully solve the source tasks but transfer substantially less effectively. Accurate propositions do not necessarily capture all decision-relevant distinctions, and states sharing the same proposition vector may require different actions.

Table 1: Ablation Study on Teacher Guidance and Semantic Representations

DomainMethodZero-shotReached 0.9T0.9 (IQM, 95% CI)AUCFinal
DoorKeyNeuSAT0.84100%1.3k [0.8, 1.8]0.981.00
DoorKeyDirect-TP0.37100%5.0k [4.8, 7.4]0.670.90
DoorKeyPPO(P)0.4279%33.8k [14.2, 88.5]0.730.84
DoorKeyDQN(P)0.1343%131.2k [124.6, 136.5]0.330.46
CrafterNeuSAT0.29100%8.5k [7.2, 10.1]0.951.00
CrafterDirect-TP0.2891%6.5k [3.5, 12.4]0.670.81
CrafterPPO(P)0.0028%147.7k [142.3, 150.0]0.200.40
CrafterDQN(P)0.005%150.0k [150.0, 150.0]0.150.24
SokobanNeuSAT0.07100%8.5k [7.2, 10.3]0.880.99
SokobanDirect-TP0.0084%45.2k [28.7, 60.0]0.520.73
SokobanPPO(P)0.0127%148.5k [145.8, 150.0]0.230.32
SokobanDQN(P)0.0028%147.3k [142.3, 150.0]0.280.43

*Metrics follow Table 1 of the paper. NeuSAT results use five evaluation seeds; Direct-TP uses five independent source seeds.

AuthorsOpenReview ↗

Clarifying the Contributions of Teacher Guidance and Semantic Representations

We thank the reviewer again for raising two important questions: how much Stage 1 teacher guidance contributes to transfer, and whether semantic representations alone explain NeuSAT's performance gains. Our new ablations attached above directly address both questions. Direct-TP demonstrates the benefit of teacher-guided distillation, while PPO(P)/DQN(P) show that identical semantic inputs alone are insufficient to achieve NeuSAT's transfer performance.

We would also like to further clarify the role of propositions in our abstraction-learning process. State abstraction learning requires both identifying how states can be grouped and determining which groupings are decision-relevant. Propositions provide a semantic bias for the former, while value information learned through interaction and Stage 1 teacher guidance determines the latter.

We have incorporated these experiments into Section 5.4 of the revised manuscript (highlighted in blue). We sincerely appreciate the reviewer's constructive feedback and welcome further discussion.

Thank you for the response, which addresses my concerns. I will update my score if necessary.

AuthorsOpenReview ↗

Thank You for Your Feedback and Continued Discussion

We sincerely thank the reviewer for their thoughtful feedback and for taking the time to consider our responses. With a few hours remaining in the discussion period, we would be very happy to clarify any further questions or concerns.

We would also greatly appreciate a score update if the reviewer feels it is appropriate. Thank you again for your constructive engagement.

Reviewer_AciXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper introduces NeuSAT which is a single-source online transfer framework that learns a visually routed state abstraction tree in a source environment and distills it into a transferable tree. this is very relevant to the offline-to-online and off-dynamics rl literature. when deployed in unseen target environments, NeuSAT transfers this propositional tree as a zero-shot prior and adapts by updating action values and refining the tree leaves with routers.

Critical strengths
  • the paper effectively shows that value-driven abstractions overfit to source layouts and distilling them into propositional rules improves zero-shot transfer.
  • strong empirical results and statistical evaluation
Critical weaknesses
  • missing related literature review:
    • examples from off-dynamics rl: [1],[2],[3]
    • examples from interpretable rl: [4], [5], [6]
  • definitions missing in section 3: Q(s,a), \omega_M (s_t), O_M,
  • in sec 3, how O_M different from S?
  • when defining the MDP M_s, how "s" is different from the state space? it seems to be a duplicate definition
  • table captions should go on top, not the bottom
  • NeuSAT is only tested on discrete, deterministic-rendering grid environments without experimenting on continuous control or partial observability.
  • the neuro symbolic distillation is not learned end to end, but it depends on an external llm with hand-crafted domain knowledge to write the perception parser and propositional vocabulary.
  • when the transferred propositional tree is insufficient in the target domain, stage 3 falls back to adding black-box visual routers rather than refining the symbolic rules which breaks the interpretability and reusability of the final policy. this contracts with the primise of NeuSAT as authors mention throughout the paper.
  • the paper provides no formal analysis or bounds on how much value-estimation error or suboptimality is introduced when compressing the source policy into bounded rules.

[1] Eysenbach et al. "Off-dynamics reinforcement learning: Training for transfer with domain classifiers." ICLR 2021.

[2] Guo et al. "Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation." NeurIPS 2024.

[3] Lyu et al. "Odrl: A benchmark for off-dynamics reinforcement learning." NeurIPS 2024.

[4] Greydanus et al. "Visualizing and understanding atari agents." ICML 2018.

[5] Koul et al. "Learning finite state representations of recurrent policy networks." ICLR 2019.

[6] Danesh et al. "Re-understanding finite-state representations of recurrent policy networks." ICML 2021.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Clarifications on NeuSAT's Learned Abstractions, LLM Usage, and Transfer Adaptation, with Expanded Related Work and Empirical Compression-Loss Analysis.

We thank the reviewer for the constructive feedback. We clarify the notation, LLM usage, and limitations of semantic abstractions, and provide additional empirical analysis of policy compression loss. We have also updated the main paper (changes highlighted in blue) with a clearer explanation of LLM-assisted proposition generation and new ablations in Section 5 demonstrating the benefits of value-guided abstraction learning and teacher-guided distillation. Detailed responses follow.

1. Related Works and Table Captions

Thank you for pointing these out. We have expanded the related work discussion to include the suggested references and moved the table captions above the tables.

2. Definitions in Section 3

We have defined Q(s,a)Q(s,a) as the expected return from taking action aa in state ss under the current policy. We clarify that S\mathcal{S} denotes the underlying MDP state space, while OMO_M denotes the RGB observation space, with ot=ωM(st)o_t=\omega_{\mathcal{M}}(s_t).

The subscripts in Ms\mathcal{M}_s and Mt\mathcal{M}_t refer to the source and target environments, respectively, rather than state or time indices. We have made these distinctions explicit in Section 3 to avoid ambiguity.

3. Neuro-Symbolic Distillation Is Not Learned End-to-End

Our goal is to learn decision-relevant state abstractions from RL interactions and express them as interpretable semantic rules for transfer. The LLM generates only the perception parser and candidate proposition vocabulary, replacing manually designed predicates used in many prior symbolic approaches. It does not determine the state partitions or policy, which are learned through value-guided abstraction and teacher-guided distillation.

This separation is a key design advantage: unlike implicit neural representations, NeuSAT explicitly identifies which states share decisions and expresses these groupings through interpretable rules. Moreover, while many existing transfer approaches rely on diverse source environments to learn generalizable representations, NeuSAT demonstrates effective transfer from a single source environment to unseen layouts and scales.

4. Transferred Propositional Tree Can Be Insufficient

As stated in Section 3, our goal is to transfer learned policies across different layouts and scales while preserving task semantics. We therefore use relational, layout-independent propositions to avoid source-specific overfitting. When propositions are insufficient in a target environment, Stage 3 continues refining the tree as in Stage 1. As shown in Table 1 of the manuscript, 100% of NeuSAT runs reach the 90% success threshold after target adaptation, even when zero-shot performance is low.

5. Value-Estimation Error and Policy Suboptimality Under Compression

While we agree that theoretical results provide additional insight, we respectfully note that the absence of general optimality guarantees does not diminish the value of algorithmic or empirical contributions. Several established RL contributions, including abstraction-related methods such as CRAR (François-Lavet et al., AAAI 2019), CURL (Laskin et al., ICML 2020), and CATRL (Dadvar et al., UAI 2023), primarily demonstrate their contributions empirically. Similarly, the original PPO and DQN papers demonstrate substantial advances without establishing general global-optimality guarantees for their learned policies. Our empirical evaluation focuses on learning neuro-symbolic abstraction trees that transfer across variations in scale and layout for similar tasks.

To directly quantify the performance impact of semantic distillation, we evaluate the source policy before and after compression. Our results show that semantic distillation is not necessarily lossless, but can preserve effective source-policy behavior while producing a transferable semantic representation. Specifically, distillation improves source return on DoorKey, while introducing moderate losses on Crafter and Sokoban. Importantly, our objective is to preserve decision-relevant distinctions for transfer rather than reproduce every source-specific value estimate.

Table 1: Source-Policy Performance Before and After Semantic Distillation

DomainStage 1 ReturnStage 2 ReturnCompression Loss
DoorKey175.97183.29-7.32
Crafter437.41400.5736.84
Sokoban71.2647.2823.98

Compression loss = J(π1)−J(πP)J(\pi_1)-J(\pi_P); negative values indicate improvement.

Thank you for the clarifications and additional ablations. However, improved adaptation performance does not resolve my concern about interpretability after adding visual routers and evaluation remains restricted to fully observed grid environments. The substantial Sokoban compression loss also warrants uncertainty estimates. Overall, I retain my score for now.

AuthorsOpenReview ↗

Clarifications on Stage-Specific Objectives, Interpretability, Partial Observability, and Compression Loss

We thank the reviewer for the response.

1. Interpretability After Adaptation

We do not claim that the final adapted policy is fully interpretable. As stated in Section 3, our objective is to efficiently learn a successful policy in the target environment. Stage 2 constructs a transferable semantic abstraction to facilitate adaptation, whose benefits are demonstrated in Table 1. Stage 3 prioritizes adaptation performance, allowing local visual refinements as in Stage 1. However, if full interpretability is desired, the Stage 3 policy could serve as a teacher for another round of semantic distillation of Stage 2.

2. Evaluation Beyond Fully Observable Environments

Our additional experiments extend the evaluation to partially observable settings. As reported in our response to Reviewer NW1E (Point 8), NeuSAT achieves 1.00 success after 50k target interactions on 64×64 Crafter maps using the official agent-centered 9×7 local observation, without additional memory.

3. Sokoban Compression Loss and Uncertainty

We would like to clarify the distinct objectives of the three stages of NeuSAT first. Stage 1 learns source policy, emphasizing sample efficiency and success rate. Stage 2 constructs a transferable semantic abstraction to accelerate Stage 3 adaptation, rather than reproduce the source policy. Preserving source-specific decisions may improve source performance but lead to overfitting and hinder transfer. Stage 3 focuses on efficient target adaptation and final policy success, rather than full interpretability.

Therefore, Stage 2's value lies in how effectively its semantic abstraction facilitates transfer. As shown in Table 2, directly transferring the Stage 1 tree yields an AUC of only 0.23 on Sokoban, compared with 0.88 after semantic distillation, under the same Stage 3 adaptation procedure. Despite the compression loss and low zero-shot success shown in Table 1, 0.07, NeuSAT reaches 90% success in every Sokoban run, with a T0.9 of 8.5k, matching Crafter and substantially outperforming the strongest baseline (51.8k). These results demonstrate that high source-policy fidelity is not necessary for effective transfer.

Why is Sokoban more affected by compression? In DoorKey and Crafter, semantic conditions such as facing a door with a key or having the crafting materials often imply consistent decisions across layouts. In Sokoban, however, the same semantic conditions may require different actions depending on the layout. For example, pushing a box toward its goal may be correct in one layout but cause a deadlock in another. Thus, Stage 2's semantic abstraction remains useful for transfer, even when its associated policy captures these layout-dependent decisions worse than Crafter and Doorkey. Stage 3 can resolve such ambiguities very fast through target-specific refinement as shown in Table 1 of our paper.

We also report uncertainty estimates for source-policy compression across five independent seeds:

DomainStage 1 ReturnStage 2 ReturnCompression Loss
DoorKey175.97 ± 22.13183.29 ± 22.46-7.32 ± 16.22
Crafter437.41 ± 23.88400.57 ± 3.9836.84 ± 20.02
Sokoban71.26 ± 8.7647.28 ± 7.1923.98 ± 7.09

*Mean ± standard deviation across five seeds.

Reviewer_NW1EOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Neuro-symbolic Abstraction and Transfer (NeuSAT), a three-stage framework for single-source online transfer in RL.

  • Stage 1 (Source Learning): The agent interacts with a single small source environment using tabular Watkins Q(λ)Q(\lambda) over discrete hashed observations and abstract tree leaves. Periodically, heterogeneous leaves are split by clustering the full concrete Q-vectors Qcon(k,⋅)Q_{\text{con}}(k, \cdot) via Ward clustering and training a local MLP router over frozen ResNet-18 features to reproduce the split.
  • Stage 2 (Semantic Distillation): Because the visually routed tree T1T_1 encodes layout-specific visual features that fail to transfer, the authors query an LLM using 6 unlabeled source frames, the domain rules/legend, and pixel statistics to synthesize a deterministic Python visual parser gg and a vocabulary of 20–50 relational Boolean propositions PP. Using offline trajectories collected by the frozen teacher (T1,Qabs,1)(T_1, Q_{\text{abs}, 1}), NeuSAT grows a new binary tree TPT_P whose internal nodes test simple Boolean combinations to minimize normalized teacher advantage dispersion, fitting leaf action values Qabs,PQ_{\text{abs}, P} via SARSA(λ)\text{SARSA}(\lambda).
  • Stage 3 (Target Transfer & Online Refinement): TPT_P and a pessimism-rebased version of Qabs,PQ_{\text{abs}, P} are transferred to larger unseen target layouts. During target interaction, leaf values are updated via Watkins' Q(λ)Q(\lambda) and heterogeneous symbolic leaves are further split locally by appending new visual ResNet-18 routers underneath them. Experiments across MiniGrid-DoorKey, a static grid puzzle rendered with Crafter sprites, and Sokoban show higher sample efficiency and transfer success compared to pixel-based PPO, JSRL, and CKA-RL.
Critical strengths
  • Full-vector Q-clustering: Splitting leaves by clustering the full action-value vector Qcon(k,⋅)Q_{\text{con}}(k, \cdot) (in Stage 1/3) and normalized advantage vectors u1(k)u_1(k) (in Stage 2) rather than scalar values of a single greedy action is a sensible improvement over CATRL/PEARL.

  • Thoughtful Stage 3 adaptation mechanics: The paper identifies two practical failure modes of static symbolic transfer: (1) source Q-values from small layouts are overly optimistic in larger target layouts (causing the agent to cycle through untried suboptimal actions), which is cleanly solved by constant-offset value rebasing (κ=0.1\kappa = 0.1); and (2) fixed symbolic leaves may remain too coarse for target-specific nuances, which is addressed by growing local visual routers beneath symbolic leaves. Figure 4 convincingly ablates both components.

Critical weaknesses
  1. Neuro-symbolic abstractions are given, not learned: The true driver of the transfer success is the fact that a strong LLM synthesizes a 100%-accurate symbolic parser with auxiliary domain knowledge provided upfront. The neuro-symbolic abstractions are not learned, they are given. If an agent is provided a 100% accurate symbolic parser and relational predicates from day one, I suspect the complex three-stage neuro-symbolic distillation pipeline is largely superfluous.
  2. Uninformative baseline comparisons: To isolate the value of NeuSAT's tree learning and distillation, the experiments need baselines that have access to the same inputs features and off-policy/tabular updates.
    • In stage 1 comparison, NeuSAT which leverages a pretrained ResNet is compared to model-free PPO baseline trained from scratch. It's well-known that off-policy/tabular value-based methods is more sample efficient than on-policy policy-gradient methods in sparse-reward deterministic grid worlds. Using a pretrained ResNet-18 and exact state hashing while restricting PPO to a randomly initialized CNN creates an unfair comparison.
    • In stages 2 & 3, a frontier LLM provided with the domain rules and legend extracts exact symbolic features for NeuSAT, whereas the transfer baselines (PPO Fine-Tuned, JSRL, CKA-RL) must learn from raw downsampled RGB pixels using a standard CNN.
  3. Unmotivated complexity in the system: There is no ablation to justify the complexity of the system. For example: maintaining a dual Q-table alongside a tree learner; combining three separate state representations (pixel-hash keys, ResNet-18 embeddings, and LLM-generated Python code); using a hand-crafted heterogeneity score with an annealing schedule for leaf splitting.
  4. Writing Clarity: The paper's exposition in Section 4 is dense and difficult to follow, partly due to the complexity of the proposed system. Critical architectural mechanics—such as how the LLM uses game rules/legends to write Python code—are buried or deferred to the appendices.
Questions for authors
  1. Since the tree T1T_1 is discard completely in the later stages, have you tried simply running standard tabular Q-learning in stage 1? How much of Stage 1's sample efficiency comes from the tree T1T_1 versus simply running tabular Watkins' Q(λ)Q(\lambda)?
  2. Since the LLM parser gg and propositions PP are synthesized from 6 random source frames prior to Stage 2, what happens if you skip T1T_1 entirely and learn TPT_P directly in Stage 1 using propositional splits?
  3. How well do PPO, DQN, or tabular Q(λ)Q(\lambda) perform on source learning and zero-shot/fine-tuned transfer when given the exact same 100%-accurate propositional vector p(o)∈{0,1}mp(o) \in \{0,1\}^m as their state input?
  4. Given that claude is provided the task rules and legend and writes a 100%-accurate symbolic scene parser, did you evaluate prompting the same LLM to write a programmatic policy directly (or act zero-shot/few-shot) over the parsed scene in the transfer setup?
  5. How can the proposed method scale to more realistic RL settings where the state space is continuous, visually noisy, or partially observable?
  6. Why was the Crafter environment scaled down from the original partially observable, dynamic 64×6464 \times 64 survival game into a fully observable, static 8×8→16×168 \times 8 \to 16 \times 16 puzzle? Does the proposed method work on the actual Crafter benchmark?
3 comments
AuthorsOpenReview ↗

Part 1/2: Clarifications on Learned Abstractions, Baseline Fairness, and System Complexity

We thank the reviewer for the detailed feedback. We address the main concerns with additional experiments comparing PPO with frozen pretrained features, PPO/DQN with identical propositions, and Direct-TP without the Stage 1 teacher. These results demonstrate that pretrained representations and semantic propositions alone do not explain NeuSAT's transfer performance, highlighting the importance of value-guided abstraction learning and teacher-guided distillation. We have updated Section 5 with these ablations and clarified LLM usage in the main paper (changes highlighted in blue). We also provide experiments on 64×64 Crafter maps and further architectural clarifications. Detailed responses follow.

1. Neuro-Symbolic Abstractions Are "Given, Not Learned"

The neuro-symbolic abstraction tree and policy are learned, not given. LLM-generated propositions describe semantic properties of states but specify neither the state abstraction (which states share actions) nor the policy (which actions should be taken). For example, two states may both satisfy carrying_key=True but require different actions depending on their surroundings. NeuSAT learns these distinctions from environmental interaction. Stage 1 uses action-value information to identify states with similar decision preferences, while Stage 2 uses the learned teacher's action preferences to select and combine propositions into a transferable semantic tree with learned action values. Thus, the LLM provides a vocabulary, whereas NeuSAT learns the actual abstraction and policy.

2. Uninformative Baseline Comparisons and Propositional Inputs to PPO/DQN

(1) Stage 1: Source Learning Comparison

Our original baselines use standard implementations (e.g., Stable-Baselines3 PPO) to evaluate NeuSAT against established RL methods rather than isolate individual components. Importantly, NeuSAT decouples visual representation from value learning. Frozen ResNet features are used for local state partitioning, while abstract Q-values are learned independently. Unlike conventional end-to-end PPO/DQN, NeuSAT does not require the encoder to directly support value prediction, but only to distinguish decision-relevant state groups.

Our additional experiments in Table 1 show that using NeuSAT's frozen pretrained ResNet cannot improve its source-learning efficiency. Replacing PPO's jointly trained CNN with the same frozen ResNet reduces learning AUC across all three domain.

Table 1: Source-Learning Performance of PPO with Trainable and Frozen Visual Encoders

DomainMethodSource StepsAUC ↑Final SR ↑
DoorKeyOriginal PPO375k0.9281.000
DoorKeyPPO + Frozen ResNet375k0.5361.000
CrafterOriginal PPO300k0.9300.917
CrafterPPO + Frozen ResNet300k0.8960.873
SokobanOriginal PPO500k0.6030.975
SokobanPPO + Frozen ResNet500k0.1090.525

(2) Stages 2 & 3: Propositional Baselines and Teacher-Guided Distillation

Our new ablations demonstrate that neither identical semantic propositions nor direct propositional learning can reproduce NeuSAT's transfer performance. PPO(P)/DQN(P) use the same parser and propositions as NeuSAT, while Direct-TP learns propositional abstractions without the Stage 1 teacher. All three underperform NeuSAT in transfer, highlighting the importance of value-guided abstraction learning and teacher-guided distillation.

We have combined these experiments into a unified ablation study in Section 5.4 of the revised manuscript. Please refer to our response to Reviewer fXJU for detailed results.

3. Unmotivated Complexity in the System

Each component addresses a distinct requirement of abstraction learning:

  1. Dual Q-tables: Abstract Q-values enable policy sharing across states, while concrete Q-values reveal within-leaf state differences that abstract values alone may conceal, guiding necessary refinements.

  2. Three representations: Pixel hashes track concrete-state values, ResNet embeddings enable visual routing, and semantic propositions replace layout-specific visual distinctions with transferable semantics.

  3. Heterogeneity score: Following PEARL (Nayyar et al., AAAI 2026), TD-error dispersion identifies inconsistent learning signals early, while value dispersion captures more stable state differences later. Annealing balances these signals as learning progresses.

Together, these components support policy sharing, decision-relevant refinement, and semantic transfer.

The remaining concerns are addressed in Part 2 of our response below.

AuthorsOpenReview ↗

Part 2/2: Clarifications on LLM Usage, Tree Abstraction, and Scalability to Realistic Settings

This is Part 2 of our response to Reviewer NW1E, continuing from Part 1 above.

4. Writing Clarity Regarding LLM Usage

We have clarified the role of the LLM in Section 5.2 (highlighted in blue).

Given high-level task descriptions, object legends, and six unlabeled source RGB frames, the LLM generates a pixel parser and candidate semantic propositions, rather than expert policies or action guidance. NeuSAT learns which propositions are decision-relevant and how to combine them through teacher-guided distillation. The LLM-generated propositions serve as candidate semantic features to guide distillation, and no further LLM calls are required during learning or transfer.

5. Comparison with Tabular Q-Learning

Although T1 is not retained after distillation, it serves two important purposes: improving source learning through value sharing across abstract states and providing a learned teacher for Stage 2.

In contrast, tabular Q-learning estimates values independently for each concrete state, making learning less sample-efficient as the state space grows. The advantage of adaptive abstraction over tabular Q-learning has been demonstrated by CATRL (Dadvar et al., UAI 2023), on which our Stage 1 builds. NeuSAT extends this approach to visual observations through learned routers and full Q-vector-based splitting.

6. Using an LLM to Write a Programmatic Policy Directly

Directly generating a reliable policy with an LLM requires knowledge of environment dynamics, which our high-level task descriptions do not provide. For example, the LLM may not know the effects of manipulating trees or stones, or how to handle stochastic transitions. Acquiring this knowledge requires environmental interaction, bringing us back to the RL setting. NeuSAT instead learns policies directly from interaction without requiring an explicit LLM-generated dynamics model or policy.

7. Scalability to More Realistic RL Settings

Discrete, fully observable environments without additional visual noise remain prominent settings in RL research, as illustrated by CROP (Altmann et al., IJCAI 2023), Feng et al. (IJCAI 2020), Alikhasi and Lelis (ICLR 2024, ComboGrid), and Taufeeque et al. (ICLR 2026, Sokoban). We further discuss NeuSAT's applicability to other RL settings below.

Continuous states: CATRL (Dadvar et al., UAI 2023), which inspires our Stage 1, demonstrates how continuous state spaces can be adaptively discretized into abstract states through value-guided refinement, with experiments on Water World and Mountain Car.

Noisy observations: Visual noise robustness is primarily a representation-learning challenge rather than our research focus. Similar to how methods such as DrAC improve PPO's robustness through data augmentation and regularization, NeuSAT can incorporate noise-robust visual encoders. Its decoupled architecture allows encoder improvements without changing the abstract value-learning mechanism.

Partial observability: Our additional experiments on 64×64 Crafter maps using the official 9×7 local observation achieve 1.00 success after 50k target interactions, demonstrating effective adaptation under partial observability in the evaluated task.

8. Original 64×64 Crafter Environment

NeuSAT achieves 100% source success and improves target success from 14% to 100% within 50k interactions on 64×64 partially observable Crafter maps, demonstrating effective transfer to larger environments.

We used compact Crafter maps in the original experiments to maintain comparable environment scales across domains. To evaluate scalability and partial observability, we additionally tested NeuSAT on 64×64 Crafter maps using the official agent-centered 9×7 observation, without additional memory.

Table 2: NeuSAT Performance on 64×64 Partially Observable Crafter

Stage 1 (Source)Stage 2 (Source)Zero-shotFinal
1.001.000.141.00

Success rates on 64×64 partially observable Crafter. Source: 5 training seeds and 1 distillation checkpoint; target: 15 held-out layouts × 3 seeds, with 50k adaptation steps.

AuthorsOpenReview ↗

Clarifying the Core Design and Contribution of NeuSAT

We thank again the reviewer for the detailed questions. We would like to further clarify the central motivation and design choices of NeuSAT, which also connect our responses above.

  1. Research objective: NeuSAT focuses on single-source transfer, unlike approaches that rely on diverse source environments to learn transferable representations. Our goal is to learn reusable, semantically structured abstractions from a source policy that accelerate target adaptation.

  2. Distinct roles of the three stages: Stage 1 learns an effective source policy from which Stage 2 reconstructs transferable semantic abstraction. Stage 3 uses this abstraction to accelerate target learning. Our added experiments support the contribution and necessity of each stage.

  3. Role of the LLM: The LLM only generates candidate semantic propositions and a pixel parser. These propositions bias the formation of the abstraction toward transferable semantic structure, while interaction-derived value information and guidance from the Stage 1 teacher determine which distinctions are decision-relevant and how states are grouped.

  4. Methodological choices and scope: The components of NeuSAT serve different purposes in value sharing, abstraction refinement, and semantic transfer. We have clarified these roles, strengthened comparisons that isolate their contributions, and discussed the current framework's limitations and possible extensions to more realistic settings.

Together, these clarifications emphasize that NeuSAT's central contribution is learning transferable semantic abstractions from a single source policy and using them to accelerate adaptation in unseen environments.

We appreciate the reviewer's thoughtful concerns and welcome further discussion to clarify any remaining questions about our approach and its contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.