acceptodds
Withdrawn from ICLR 2027

Morphosemantic Concept Systems: Learning Reusable Discrete Concepts for Compositional Generalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human languages are characterized by the ability to abstract recurring meanings into reusable concepts and compose them into novel expressions beyond previously observed forms. However, in conventional large language models (LLMs), reusable semantic concepts remain implicit in surface-level tokens and continuous representations, making them difficult to explicitly reuse and systematically recombine in novel contexts. We introduce Morphosemantic Concept Systems (MCS), an intermediate machine language that explicitly represents concepts as discrete identities with a growing inventory, enabling concepts to be reused and recursively composed across different linguistic forms. MCS employs two learnable translators to bridge human and machine languages: an input translator maps natural language into concept-level representations, while an output translator converts generated concepts back into human-readable forms; during training, a gradually annealed teacher provides early contextual supervision to bootstrap concept formation and is removed during inference. Experiments demonstrate that MCS achieves competitive performance with limited training resources. By effectively exploiting and recombining the inherent semantic relationships within sentences through the proposed concept-based representation, MCS validates the potential of resource-efficient semantic abstraction for robust structured reasoning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
compositional generalization, discrete representations, concept learning, hierarchical composition, tokenization, semantic parsing, neurosymbolic, language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Lw5tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose to represent words as bundles of conceptual features rather than as atomic units. This is hypothesized models' ability to generalize compositionally, presumably because the conceptual features can generalize even when the full word doesn't. The authors show that this method is effective on a number of synthetic generalization tasks.

Critical strengths

None.

Critical weaknesses
  1. No insight is provided into what the model learns and why it works. It is unclear what the "concepts" that the model ends up learning are, and how they help generalization. The comparison to tokenization papers implies that the authors think of the model as a sort of morphological decomposition (a more linguistically relevant tokenization than just BPE), but I don't understand why that should improve structural generalization in a dataset like SLOG. There is no error analysis or comparative analysis of models.

  2. No attempt is made to compare the model to earlier models (there are empirical experimental comparisons but no discussion of the conceptual differences). Some earlier models are cited but it's not clear how they're related to the current model so it is hard to evaluate novelty. An obvious simple baseline here would be to apply a morphological decomposition and provide the output as the tokens, instead of learning a tokenization algorithm.

Questions for authors

None.

Reviewer_zRVmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether explicit, reusable discrete concepts can help language models generalize to unseen compositions. It proposes Morphosemantic Concept Systems (MCS), an intermediate machine language in which each word is represented as a glyph. Each glyph is composed of concepts from a shared inventory, with a transcription pathway for forms such as proper names. Learned translators map text into glyphs and back, and a small causal LM operates over the glyph sequence. The concept inventory grows during training, and an annealed teacher gives binary accept/reject feedback on new glyphs in early training. Small Transformers trained from scratch are evaluated on CFQ, COGS, and SLOG against alternative tokenization and sequence-unit methods. The paper reports that MCS achieves competitive or better performance than these baselines with a small model.

Critical strengths
  • Representing words as explicit compositions of shared discrete concepts is a reasonable direction for compositional generalization.

  • The paper uses three established compositional generalization benchmarks and reports several complementary metrics beyond exact match.

  • The ablations target the main design choices, such as shared versus separate identifiers and the conceptual versus transcription pathway.

Critical weaknesses
  • The core method is under-specified, which makes the results hard to interpret or reproduce. The paper describes only at a high level how concepts are predicted for a word and when a new concept is added to the inventory. The training objectives are only named and never defined, and the teacher used in the experiments is not specified. The Reproducibility Statement refers to an appendix with full details, but the submission seems not to provide it. The paper also doesn't include an analysis of the learned concept inventory, so there is no direct evidence that the concepts are reusable or semantically meaningful.

  • The role of the teacher weakens the main claim. Section 3.4 states that the teacher injects no semantic target information. However, a pretrained LM or human that accepts or rejects glyphs based on whether they are "semantically reasonable" is an external semantic signal, and Section 4.1.3 describes it as a "teacher prior" used "for concept initialization". Without the teacher, EM drops from 0.7712 to 0.5811 (Table 7), close to the token-based variant (0.5527 in Table 4). Most of the gain from the concept representation therefore appears to depend on this external supervision.

  • The baseline comparison does not clearly support the claim of improved compositional generalization. The baselines are tokenization and sequence-unit methods, not methods designed for compositional generalization. Except for H-Net, all baselines get zero or near-zero strict EM on all three benchmarks, which is well below the seq2seq baselines reported in the original benchmark papers. The paper does not explain how the baselines were trained for these tasks. It also does not compare with existing methods designed for these benchmarks. It is therefore unclear whether MCS is stronger than a simple, properly tuned baseline.

  • There seem to be some inconsistencies in the reported results. The full MCS model has different edit distances across the ablation and sensitivity tables (13.3141 in Table 4 and 13.3457 in Tables 5-9). Some metrics are also reported on mixed scales within the same table (Tables 6-7). These issues make it harder to rely on the ablation results.

Questions for authors
  • Which teacher is used in the experiments, and how much does it contribute relative to the concept representation itself?

  • How were the baselines trained on these semantic parsing tasks, and why do most of them get zero or near-zero strict EM?

Reviewer_EkchOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors argue that conventional language models represent semantic concepts implicitly. This may limit their ability to combine familiar concepts in new ways. To address this, they propose Morphosemantic Concept Systems (MCS). It represents words as glyphs composed of reusable discrete concepts. The method uses a growing concept inventory and two learned translators to convert between natural language and glyphs. A causal language model predicts glyph sequences. A teacher provides early feedback to guide concept construction, which is removed during inference. Experiments on CFQ, COGS, and SLOG show that MCS achieves higher relation or structural F1 among the compared methods. Further ablations examine the contributions of the proposed components.

Critical strengths

The proposed approach to compositional generalization is interesting and well motivated. Prior work explored concept-based and intermediate representations. MCS brings these ideas to compositional generalization through reusable discrete concepts, shared identities, and a growing inventory. The dynamic concept inventory and the learned concepts could also serve as a useful resource.

The experimental results are generally positive. MCS achieves competitive results on three compositional generalization benchmarks, especially on structural metrics, with relatively few parameters. The ablation and sensitivity studies also help explain the contributions of the main components.

Critical weaknesses

The concept learning process and its benefits are not well explained or validated. Although the idea is well motivated, the paper does not clearly explain how concepts are discovered, named, and assigned during training. It is also unclear how the concept assignments used to train the translators are obtained and updated. The results and ablations do not demonstrate that the learned concepts are meaningful, reusable, or support systematic generalization.

The proposed method depends on teacher supervision and two learned translators. However, the teacher implementation is not clear. It appears that removing the teacher causes a large performance drop. The H2M and M2H translators may introduce additional errors. Their individual contributions and the effects of error propagation are not evaluated separately.

The related work and baseline comparisons are incomplete. The paper should also discuss prior work in compositional generation with citations. The paper mainly compares methods for alternative tokenization and sequence representations. Several relevant approaches to compositional generalization are missing.

The scalability and computational cost are not sufficiently studied. While it shows that increasing model capacity reduces performance, this could be caused by overfitting or insufficient training. The paper does not investigate the reasons beyond the numbers reported. The authors also do not report the training cost of dynamic concept construction, teacher supervision, and the two translators. Although multiple seeds are mentioned, the main results do not report variance or statistical significance.

[1] Guo et al. Hierarchical Poset Decoding for Compositional Generalization in Language.

[2] Shi et al. Revisit Systematic Generalization via Meaningful Learning.

[3] Herzig et al. Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations.

[4] Petit et al. Structural Generalization in COGS: Supertagging Is All You Need.

Questions for authors

How are new concepts discovered, named, and assigned during training?

Is there any evidence that the learned concepts have consistent meanings and can be reused in unseen compositions?

Is the learned concept inventory reusable?

How does MCS compare with existing compositional generalization methods like HPD, LeAR, etc. ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.