acceptodds
Under review as a conference paper at ICLR 2027

Neural-Symbolic Representation Learning by Structural Abstraction from Noisy Demonstrations

Abstract

How can neural perception be interfaced with symbolic reasoning when primitive symbols and their structure are not given in advance? Palaeographers infer how oracle-bone characters are composed by analysing their shapes and possible strokes, which are noisy demonstrations composed by a sequence of raw perceptual observations without any symbolic labels or predefined structures. In this case, the learner must abstract both the primitive symbols and infer how they compose, unlike traditional neural-symbolic (NeSy) learning tasks which usually assume a manually specified symbols of primitives and relations as the interface. We study this representation learning problem and introduce structural abstraction, an algorithm for learning the interface from such demonstrations. On few-shot handwritten oracle-bone characters, it recovers expert-annotated radical substructures with 72.28% macro-averaged recall, compared with 9.58–13.09% for feature-based motif-discovery baselines. The learned representations extrapolate to unseen ancient and modern Chinese characters and discovers novel radicals at the same time. Our results show that perceptually grounded symbols and their compositional structure can be learned from demonstrations.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neural-Symbolic Representation Learning, Structural Abstraction, Substructure Discovery, Oracle Bone Character Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mqeYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Structural Abstraction, a framework for jointly learning neural predicates, reusable substructures, and formation rules from noisy perceptual demonstrations. The method combines variational predicate discovery, structural consistency search over contiguous subsequences, constrained grammar induction, and iterative representation refinement through abductive reasoning. Experiments on oracle-bone characters, modern Chinese characters, unseen radicals, and controlled digit sequences demonstrate substantially higher substructure discovery recall (SDR) than three feature-based motif-discovery baselines, along with generalization to unseen characters and radicals.

Critical strengths
  1. Valuable problem formulation. Unlike conventional neural-symbolic approaches that assume a predefined symbolic vocabulary, this work treats predicate identities, part boundaries, and compositional rules as joint learning targets. This formulation highlights an important challenge in learning the interface between neural perception and symbolic reasoning.
  2. Reasonable integration of perceptual and structural learning. The iterative framework allows structural consistency and induced rules to refine perceptual predicate assignments, while improved predicate representations facilitate the discovery of recurring substructures. Experiments on oracle-bone characters, modern Chinese characters, and controlled digit sequences provide complementary evidence across noisy real-world observations and settings with verifiable underlying grammars.
Critical weaknesses
  1. Limited evaluation of substructure discovery quality. The primary metric is macro-averaged recall over annotated substructures, which does not adequately assess false discoveries or the complexity of the induced symbolic theory. Since the method may generate numerous candidate substructures, reporting recall alone could overestimate discovery quality without complementary precision-oriented metrics or analysis of the learned theory size.
  2. Heavy reliance on predefined assumptions and hyperparameters. The framework depends on several manually specified components, including similarity thresholds, initial numbers of candidate symbols, pre-segmented strokes, known stroke orders, predefined relation types, a fixed abstraction hierarchy, and grammar induction templates. These assumptions substantially constrain the search space and raise concerns about the method's generalizability to less structured perceptual data, where such prior knowledge may not be readily available.
Questions for authors

Please refer to Weaknesses.

1 comment
AuthorsOpenReview ↗

Author Rebuttal to Reviewer mqeY

We thank the reviewer for recognising the problem formulation and the interaction between perceptual and structural learning. We address discovery quality and prior assumptions below.

Precision and theory complexity

Response: We thank the reviewer for pointing out this evaluation gap. We agree that recall alone does not assess false discoveries. We have therefore added precision measurements:

DatasetAE+LoCoMotifSimCLR+LoCoMotifVBM+LoCoMotifOurs
Oracle-bone characters11.13%11.06%20.52%63.25%
Modern Chinese characters59.62%58.53%61.54%77.78%
Controlled digit sequences26.03%18.18%27.09%100%

These results complement SDR. We describe the measure as annotation precision because expert annotations are incomplete. For example, the radical “足” contains the structure “口”, but this nested component is absent from the annotations. An unmatched part may therefore be meaningful despite being counted as a false discovery under the annotation scheme. We will clarify the precision calculation and its limitations.

We also report the learned predicate counts, rule counts and part occurrences from the existing runs:

DatasetAtomic PredicatesPart PredicatesWhole PredicatesWhole-level RulesPart-level RulesPart Occurrences
Oracle-bone characters933612822750016,545
Modern Chinese characters152151,0001,00020121,937
Controlled digit sequences10127376717,217

These counts describe the size of the learned theory. They include abstractions beyond the annotated radicals. Additional symbols and rules may also accommodate perceptual noise, as illustrated by the learned digit grammar. We will clarify the counting conventions and relate these statistics to the theory-size penalty in Eq. 1.

Sequential assumptions

Response: We thank the reviewer for raising this concern. We suggest that segmented atomic observations are common, for example in imitation learning and human's everyday teaching.

For example, a higher-order task decomposition of making coffee involves grinding beans, heating water and brewing. However, for a learner, a demonstration may record only sensory observations and low-level actions, such as visual inputs and muscle movements. Hence, each transition xt=(st,at,st+1)x_t=(s_t,a_t,s_{t+1}) is such an atomic observation. Its temporal position is given, but the learner is not told which transitions belong to what high-level behaviour or how the behaviours compose. Given a fragment of an ordered raw trace,

+--------+   a0   +--------+   a1   +--------+   a2   +--------+
| raw s0 |------->| raw s1 |------->| raw s2 |------->| raw s3 |
+--------+        +--------+        +--------+        +--------+

a high-level symbolic description might be a grammar for example a regular expression A1+A2+A3+…A_1+A_2+A_3+\ldots. However, the symbols and their grounding must be learned:

  • Symbol invention: the set of abstracted symbols Σ={A1,A2,A3,…}\Sigma=\lbrace A_1,A_2,A_3,\ldots \rbrace grows as concepts are discovered.
  • Perceptual grounding: ϕk:X→[0,1]\phi_k:X\to[0,1] scores whether an observation in raw sensory feature space XX belongs to abstracted concept AkA_k.
  • Relational hypotheses: symbolic rules such as A3←A1∧A2A_3\leftarrow A_1\wedge A_2 (A3A_3 after when A1A_1 and A2A_2 hold).

This motivates our setting, and the abstraction is exactly the target of interface learning.

About hyperparameters

Response: We thank the reviewer for raising this concern. Most hyperparameters were chosen heuristically, with limited tuning. We agree that studying their effect on performance is valuable. Sensitivity experiments are running, and we will report their results in the revised paper.

Reviewer_vpmWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to identify the corresponding symbolic labels and the underlying rules from raw perceptual observations without given symbolic labels or predefined structures. Taking oracle bone inscriptions as an example, the initial input consists only of an overall image of an inscription and a sequence of sub-images on that image (representing individual strokes, with the sequence order being a manually inferred stroke order). The authors aim to map each sub-image to an atomic label and to identify common patterns formed by some sub-images (called parts in the paper). After this identification, they summarize rules, such as which atomic labels constitute a part and which parts and atomic labels constitute an overall inscription image. The paper formalizes this problem and proposes a structural abstraction algorithm to solve it. Experiments show that this method can indeed find parts already discovered by humans across various datasets.

Critical strengths

This paper considers a very interesting problem. Inferring the atomic labels, parts, and corresponding rules from a noisy image is indeed a challenging and worthwhile research problem, and it is of great help to the broader field of interpretable learning. The algorithm proposed in the paper is intuitive, and the part discovery results show a reasonable recall rate.

Critical weaknesses

My current overall stance is slightly positive. However, I have identified two weaknesses, which I will also detail in the questions below; whether I maintain my positive stance will depend on the rebuttal.

One weakness is that some key details of the algorithm are unclear in the paper. The overall framework of the paper and the algorithm is relatively clear, but some details of the framework are not well stated. Section 3.3 is somewhat hard to follow; after reading it, I do not understand how the rules are obtained. The paper only says that the MIL algorithm is used but does not provide its details. The notion of “hierarchical structural consistency” in Lines 279–281 is also unclear: what it means and what it is used for. Section 3.3 appears to be a critical step, but it receives little description. By the way, in Section 3.1, the paper already mentions using VBM to classify ℓ=part\ell = part, which was confusing at first, especially since Algorithm 1 appears much later; initially I did not know that the first step of the algorithm only operates at the atomic and whole levels.

Another weakness is the experimental section. The experiments do not evaluate the rules induced for oracle-bone characters and modern Chinese characters; they only report a recall rate based on statistics of the returned parts. Moreover, the paper uses LoCoMotif as a baseline, but LoCoMotif can only find corresponding parts when the stroke order is the same across characters, whereas the proposed method does not depend on stroke order and can identify a part as long as its strokes are contiguous, even if their order differs across character inputs. I am unsure whether this affects the comparison with the baseline; this point needs to be discussed and clarified.

Questions for authors

(1) Could you explain Section 3.3 in more detail? How exactly is the “hierarchical structural consistency” mentioned in Lines 279–281 defined, and what is its purpose? How does the MIL algorithm used there work specifically?

(2) What is the recall of the rules induced for oracle-bone characters and modern Chinese characters relative to the rules that have already been discovered?

3 comments
AuthorsOpenReview ↗

Author Rebuttal to Reviewer vpmW

We thank the reviewer for the positive assessment and constructive comments. We clarify rule induction, HSC and the scope of the experimental evaluation below.

HSC and MIL in rule induction

Response: We thank the reviewer for identifying these missing details. Meta-rules define the permitted clause forms. The MIL procedure induces clauses within this hypothesis space to explain recurring compositions. Candidate rule patterns are initialised or carried over from the previous iteration. Demonstrations are grouped by HSC, their symbolic descriptions are used to construct or revise clauses, and compatible patterns are merged. Part-level clauses define parts through atomic predicates and relations. Whole-level clauses describe compositions of parts or atomic observations.

HSC assesses compatibility across whole, part and atomic descriptions. It guides demonstration grouping and pattern merging. Initially, only whole and atomic predicates are available; part predicates are learned after recurring segments are discovered. We will explain this order earlier in Section 3.1. We will also expand Section 3.3 and the appendix with the HSC aggregation, alignment procedure and rule induction steps, supported by a worked example.

Rule evaluation and LoCoMotif comparison

Response: We thank the reviewer for highlighting these evaluation concerns. SDR measures the recovery of annotated radical instances. Character-level rule recall would require reference rules specifying the stroke predicates and compositional relations that define each character. The available annotations identify radical categories and instances, but do not provide these complete rules. We therefore cannot report reliable character-level rule recall without further expert annotation. We will state this evaluation gap explicitly. The digit experiment checks induced rules against a known grammar. Its reported 100% consistency applies to returned parses: 84.64% of test sequences are fully parsed, 13.67% are partially parsed and 1.69% remain unparsed. These results provide controlled evidence of rule correctness, rather than character-level rule recall.

LoCoMotif is a feature-space motif-discovery baseline, rather than a direct competitor for the full NeSy representation-learning task. It provides a reference for recovering recurring annotated parts from AE, SimCLR and VBM features. VBM + LoCoMotif is particularly informative because it uses the same type of perceptual model as our method, but discovers parts through feature-space motifs. Our method jointly learns grounded predicates, reusable parts and composition rules through structural reasoning. The comparison evaluates the shared objective of annotated part recovery. As stated in the paper, annotation information is also used to merge LoCoMotif groups for evaluation.

Thanks to the authors for their response. I will maintain my initial positive rating.

We sincerely thank the reviewer for maintaining the positive rating and for the continued encouragement.

Reviewer_JECPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper targets the problem of how neural perception can be linked with symbolic reasoning when the symbols and their structure is not known. The problem setting is novel as existing work either considers the symbols to be known or doesn't use neural perception. The paper considers a case where oracle-bone characters need to be studied in terms of their primitive strokes, the part hierarchy over the strokes and then the placement of the parts to form the character. The approach paper provides is that of first starting with a theory L the symbols are inferred by the variational approximation of posterior. These symbols are used by abductive reasoners which update the prior. The inference model is then trained again using the ELBO (as prior is updated) and the process is repeated until the theory and assignment is stabilized. The method though assumes the part relations and also the restriction of the grammar to be given. The results on oracle-bone characters, Chinese characters and controlled digit sequences shows the proposed method can discover the substructures more effectively than baselines. The learned structure also transfers to unseen characters.

Critical strengths
  1. The paper is one of the first works to target the learning of symbolic theory and neural perception jointly from given demonstrations (to the best of knowledge of the reviewer).
  2. The proposed method seems intuitive, and is well grounded in theoretical concepts of latent space models.
  3. The results support that the proposed method is able to recover the substructure from the three noisy character datasets.
Critical weaknesses
  1. The paper assumes a specific structure on sub-parts, i.e., the corresponding atomic units are provided sequentially in the demonstration. While this curtails the search space dramatically, this may limit the kind of demonstrations that the approach can handle.

  2. The access to the primitive strokes and the part reactional knowledge also takes the setting in the restricted domain where these things are known. Specifically, assuming knowledge of relationships seems limiting, since that also constraints the kind of predicates that can be learned.

  3. The optimization process is pretty complicated, and time consuming. Authors do not report any timing experiment for learning/inference.

  4. The writing of the paper can be improved a lot. Specifically the writing can be simplified potentially by reducing the number of symbols used wherever possible, and providing more intuitions. Examples could be included in between the method sections. The high level summary of the method can be written at the top of the method.

    a. It is not clear how priors are updated (line 293-294)

    b. The user of structural consistency score, though intuitive, is not precisely described. Is it computed for every such pair of sub-parts with matching sequence lengths? Does that not blow up the computation? And how are these scores aggregated?

Questions for authors
  1. More intuition in Eq 3 (before ELBO).
  2. Adding more details on the algorithm would benefit the understanding of the reader. Specifically how the abductive reasoning is updating the prior. The details on the variational inference.
  3. Is it possible to learn the part relations also without the need of the relational knowledge?
  4. Can you test the method on some more visually complex dataset? Like the example of the two wheeled scooter from [1].
  5. Please refer to the weakness section for further questions.
  6. How does the work compare with earlier paper by Tennanbaum on learning programs for similar tasks [2]? Is a head to head comparison possible between two techniques? It will be good to include discussion
  7. The method explores a fixed 3 depth predicate structure where parts are formed by primitive predicates. It will be good to know authors’ thoughts on how the approach might handle the general case where the hierarchy level is more than three, or even extending to the case where the number of levels is unknown (and is more than three).

Reference: [1] Brenden M. Lake et. al., Building Machines That Learn and Think Like People, 2016 (https://arxiv.org/pdf/1604.00289) [2] Brenden M. Lake et. al., Human-level concept learning through probabilistic program induction, 2015 (https://www.science.org/doi/10.1126/science.aab3050?referrer=https%3A%2F%2Fwww.google.com%2F)

4 comments
AuthorsOpenReview ↗

Author Rebuttal to Reviewer JECP

We thank the reviewer for the careful reading and constructive comments. We appreciate the recognition of our problem formulation. Below, we clarify the supplied assumptions, the learned representations and the learning procedure. We will include these clarifications and the additional results in the revised paper.

Sequential observations and relational knowledge

Response: We thank the reviewer for highlighting these assumptions. We suggest that segmented atomic observations are common, for example in imitation learning and human's everyday teaching. For the motivation and an example, please see Sequential assumptions in our response to Reviewer mqeY.

Our method learns relational knowledge, including which strokes and spatial relations define a radical and how radicals form a character. Basic spatial predicates and their computation programs are supplied as background knowledge. They provide the shared relational language needed to express reusable composition rules and are necessary for the concept learning task. We will clarify this distinction in the revised paper.

Computational cost

Response: We thank the reviewer for noting the missing runtime results. We have measured training and inference times for all three experimental settings. The table separates neural predicate learning, grammar induction and subsequence discovery. Total training time is accumulated over the reported epochs.

DatasetEpochsNeural predicate learning (s/epoch)Grammar induction (s/epoch)Subsequence discovery (s/epoch)Total training time (s)
Oracle-bone characters5056.55105.050.0718083.47
Modern Chinese characters20202.54282.2036.7910430.93
Controlled digit sequences50125.10529.640.0732740.22

The following inference times cover the reported test sets.

DatasetNumber of test samplesInference time (s)
Oracle-bone characters2403.44
Modern Chinese characters1301.07
Controlled digit sequences15431.79

Clarity of the learning and refinement procedure

Response: We thank the reviewer for pointing out this lack of clarity. We will revise the main paper and appendix to explain HSC, rule induction, substructure matching and abductive prior updates, with a worked example.

Intuition for Eq. 3

Response: We thank the reviewer for requesting this clarification. The KL divergence measures the mismatch between an observation's embedding distribution and a candidate predicate's embedding distribution. Its negative exponential gives a perceptual compatibility score: a smaller divergence gives a higher score. Eq. 3 combines this score with the symbolic prior p(c∣h(Σ))p(c\mid h(\Sigma)), then normalises over candidate predicates. Thus, predicate assignment reflects both perceptual evidence and structural constraints. At test time, we use a uniform symbolic prior.

Abductive prior and variational inference

Response: We thank the reviewer for highlighting this unclear step. Abduction finds predicate assignments that allow demonstrations to share a compositional explanation, and uses them to construct the symbolic prior for the next neural update. Variational inference approximates the intractable posterior over predicate assignments and latent embeddings by maximising the ELBO. The resulting assignment probabilities combine perceptual compatibility with the theory-dependent prior.

More complex observations and deeper hierarchies

Response: We thank the reviewer for raising these questions. Our task combines perceptual variation with the difficulty of discovering symbols and rules from few demonstrations. The current method uses three levels. More complex scenes and unknown hierarchy depths remain future work, potentially guided by prior knowledge from pre-trained language and vision-language models.

Relation to Bayesian Program Learning

Response: We thank the reviewer for pointing out this related work. We will cite Lake et al. (2015) and discuss BPL in the revised paper. BPL learns probabilistic generative programs for handwritten concepts to support tasks such as one-shot recognition, generation and parsing. Our task jointly discovers grounded neural predicates, reusable parts and first-order composition rules; a direct numerical comparison requires a common task and evaluation protocol.

AuthorsOpenReview ↗

About sequential observations

We thank the reviewer for highlighting this assumption, and would like to hear about your opinion before the rebuttal phase ends.

We distinguish atomic observations from their symbolic identities. The learner receives individual stroke observations and their order. It must discover which strokes are variants of the same primitive, which strokes form reusable parts, and how these parts compose into wholes. Sequential organisation is an inductive bias on the input. It does not specify the learned predicate vocabulary or composition rules.

Sequential demonstrations also occur in imitation learning and everyday teaching. For example, making coffee involves grinding beans, heating water and brewing coffee. A demonstration records sensory observations and low-level actions, such as reaching, grasping and pouring. Each transition xt=(st,at,st+1)x_t=(s_t,a_t,s_{t+1}) can serve as an atomic observation. Here, sts_t is a raw state observation and ata_t is a recorded low-level action. The transition boundaries and temporal order are given. The learner must still discover abstract behaviours, their boundaries and their composition rules. For example, ground coffee and hot water must be ready before brewing:

Given a fragment of an ordered raw trace

             a0                a1                a2
+--------+        +--------+        +--------+        +--------+
| raw s0 |------->| raw s1 |------->| raw s2 |------->| raw s3 |
+--------+        +--------+        +--------+        +--------+

The atomic observations are transition tuples xt=(st,at,st+1)x_t=(s_t,a_t,s_{t+1}). Their temporal order is given, but their symbolic identities and reusable segment boundaries are unknown.

For illustration, a learned trace model might be a∗b∗c∗a^{*}b^{*}c^{*}. Here, ∗* means zero or more repetitions. If all three stages are required, as in the coffee example, the pattern would be a+b+c+a^{+}b^{+}c^{+}, where ++ means one or more repetitions.

  • Perceptual grounding: ϕk:X→[0,1]\phi_k:\mathcal{X}\to[0,1] scores whether a raw transition xt∈Xx_t\in\mathcal{X} belongs to a learned predicate kk.
  • Symbol invention: the learned alphabet Σ={a,b,c,…}\Sigma=\{a,b,c,\ldots\} can grow as new concepts are discovered.
  • Relational hypotheses: schematic rules such as c←a∧bc\leftarrow a\land b, with arguments and specific relations left implicit.
  • Behaviour execution: a future extension would learn policies πk(at∣st)\pi_k(a_t\mid s_t) and termination probabilities βk(st)\beta_k(s_t) for the discovered behaviours.

The recorded actions ata_t are inputs; the abstract symbols a,b,ca,b,c are learned. Combining abduction, induction and neural learning is a possible route to these abstractions. Behaviour execution remains future work.

Our current character experiments assume pre-segmented strokes and search for parts within contiguous stroke segments. Strokes within a segment may be aligned in different orders. Thus, the method learns part boundaries within the supplied sequence, while atomic segmentation is given.

Thanks authors for response. My concerns are partly addressed, specifically on weakness 1, weakness 3 and Q1, Q6. But my main concerns are still open. I think the reliance on type of relational predicates, different depth generalization, and also the hyperparmetes (as also mentioned by the reviwer mqeY) makes method less generalizable to different domains.

We thank the reviewer for raising these concerns. We would like to emphasize that, to the best of our knowledge, no existing method can reliably discover meaningful symbolic abstractions from few-shot, unlabelled demonstrations without relying on any form of prior knowledge or inductive bias. This is particularly challenging because a limited number of observations alone is insufficient to uniquely identify the underlying symbolic structures. Background knowledge is therefore essential, rather than an arbitrary restriction, for making symbolic abstraction learnable in such a setting.

We argue that relations are fundamental to compositional generalization: without knowing how individual primitives are related, it is difficult to discover reusable parts and generalize their compositions to unseen structures. Importantly, we only assume basic spatial relations, which constitute the most fundamental relational knowledge for describing how visual primitives are organized. We do not provide the symbolic identities of primitives, reusable parts, or their composition rules; these are precisely what our method aims to discover from demonstrations. We acknowledge that learning the relational vocabulary itself is an interesting extension, but consider it a distinct and substantially more challenging problem.

The current three-level structure is an explicit structural inductive bias tailored to our target tasks, rather than a fundamental limitation of the framework. The compositional nature of our framework allows it to be extended to deeper hierarchies by recursively composing previously learned abstractions into higher-level concepts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.