- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies out-of-composition (OOC) generalization, where a model solves previously unseen tasks by recombining learned operations in ways that differ from the compositions encountered during training.
To address this problem, the authors propose Neural Language Programming (NeuLP), a framework consisting of three stages:
- Language Acquisition: The model jointly learns a discrete neural language, a programmer that predicts sequences of operations, and a differentiable interpreter that executes them. A program-space self-curriculum progressively increases the allowed program length, while a Bayesian execution-evidence prior (BEEP) encourages the discovery of reliable and reusable operations.
- Compositional Optimization: With the learned modules frozen, the method employs execution-guided discrete search to discover successful programs for unseen compositions. It also introduces task-free dreaming to generate additional synthetic programs and input-output examples.
- Learning from Discovered Programs: The programmer is further trained using the programs discovered in the previous stage, improving its initial program proposals and reducing the need for extensive search. Overall, the framework is designed to first acquire a reusable neural language and subsequently improve its ability to compose the learned operations for OOC generalization.
The authors evaluate NeuLP on four newly proposed compositional benchmarks: Comp-II, CompARC, CompCIFAR, and CompLife. These benchmarks evaluate generalization to unseen recombinations and longer compositions of learned operations. The experimental results demonstrate substantial improvements over existing neural program synthesis approaches, including LPN and NLI.
Critical strengths
- Clear motivation and well-presented methodology.
The paper is well written and interesting to read. The motivation is particularly compelling, with a clear discussion of the relevant literature and the challenges associated with compositional generalization.
I also appreciate how the methodological choices are closely connected to the identified challenges. Although NeuLP involves a relatively complex three-stage training procedure, each stage serves a clearly motivated purpose, making the overall framework coherent and understandable.
The figures substantially help readers understand the proposed framework, and the extended discussion of related work in the appendix further improves the clarity and readability of the paper.
- Meaningful problem formulation and benchmark contributions.
The paper addresses OOC generalization, which I consider an important problem deserving further attention. The ability to reuse and compose previously learned operations to solve unseen problems is a meaningful direction for developing more generalizable learning systems.
Additionally, the authors introduce four compositional benchmarks designed to evaluate different aspects of OOC generalization. The distinction between unseen recombinations and increased compositional depth provides a clear and systematic evaluation framework.
Critical weaknesses
I do not identify a clear methodological flaw in the proposed framework or its evaluation under the stated OOC setting. However, I have some reservations regarding its broader applicability and practical significance, in the questions part.
Questions for authors
- Limited demonstration of applicability beyond controlled compositional domains that can be composed with finite primitives.
The proposed framework, as well as much of the closely related neural program synthesis literature, focuses on domains where target functions can be represented as compositions of reusable operations.
The four proposed benchmarks follow this assumption, with each task generated from sequential compositions of a finite set of underlying primitive operations.
I acknowledge that this is precisely the setting that NeuLP aims to address, and that the controlled benchmarks provide a reasonable evaluation of its stated objective. Therefore, I do not regard their synthetic nature as an inherent weakness.
Nevertheless, I wonder whether the framework can be extended to more practical domains where the underlying computational structure is less explicitly defined or where discovering reusable operations is itself more challenging.
If NeuLP could demonstrate similar compositional generalization in such a domain, I believe its potential impact would be considerably greater. In particular, a practical application where OOC generalization is genuinely necessary would further strengthen the significance of the proposed approach.
Response to reviewer Jp2o
Thank you for the positive review. We also agree that broader applicability is an important next step.
Question #1. Can NeuLP extend beyond controlled compositional domains to more practical settings?
We see NeuLP as a foundation for neural language program learning in richer real-world settings, rather than a complete solution to practical world modeling in its current form.
The current version makes several simplifying choices that we already discuss in the limitations. The language (library) is a flat finite codebook, and programs are bounded sequences. Both can be extended. A richer library could support more structured operations and stronger regularization over how they are learned and reused. The program representation could also support memory and control flow. For example, branching would require explicit control flow, while memory would allow sequential programs to store and reuse intermediate values. Another option is to have the programmer predict a graph directly.
These are natural extensions of the current framework. The core idea remains the same: learn an executable language from observations, then learn to program with it. We also see practical world-modeling settings as an important direction for future work.