Learning Neuro-Symbolic Abstractions for Transfer in Reinforcement Learning

Reinforcement learning agents often require substantial interaction to adapt to a new environment, even when the new environment preserves the task semantics of a previously solved source environment. Reusing knowledge acquired in the source can reduce this cost, while doing so requires the agent to identify states that play the same decision-relevant role across environments. State abstractions provide a natural way to share action-value estimates and policies across such states, while abstractions induced by value signals from the source environment may encode layout-specific distinctions and therefore transfer poorly. We introduce Neuro-symbolic Abstraction and Transfer (NeuSAT), a single-source online transfer framework built on the observation that an abstraction effective for source learning is not necessarily suitable for transfer. NeuSAT first learns a visually routed abstraction tree and an abstract policy through interaction with the source environment. It then uses them as a teacher to rebuild the state partition with auto-generated semantic propositions and fits abstract action values on the resulting proposition-routed tree. The propositional tree and its action values are transferred to unseen target environments as a zero-shot policy prior. During the agent's interaction with the target environment, NeuSAT updates the transferred values and locally refines leaves that combine states with conflicting action preferences. Our evaluation results on three domains across held-out layouts and environment scales show that proposition-structured state abstractions can support sample-efficient transfer from a single source environment.