acceptodds
Under review as a conference paper at ICLR 2027

Inducing Unseen Programs as Transformations in a Learned Representation

Abstract

Program induction uses input–output examples to infer how an unknown pro- gram behaves on further inputs. We ask whether a shared representation of data can make this inference possible for programs absent from training. We intro- duce Joint-Embedding Program Inductive Architecture (JEPIA), which encodes a task’s input–output examples and fits a transformation between their embeddings. The fitted transformation predicts output embeddings for further inputs. Errors on these predictions train the shared encoder through the fitting procedure, so the representation is learned for the inference it must support. We instantiate JEPIA with rotations and reflections, which admit an analytical fit, and regularize the em- beddings against collapse. In modular addition, digit embeddings acquire a ring structure in numerical order, providing an interpretation of how the model pre- dicts held-out additions. On cube transformations, the model transfers from two- move training programs to programs generated by shorter and longer sequences. Additional context examples and broader exposure to training programs improve held-out prediction. These results support learning data representations that allow a common fitting procedure to infer new programs from examples.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
program induction, representation learning, compositional generalization, joint-embedding predictive architectures, meta-learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ctQsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes JEPIA, a JEPA-style approach to learning representations of programs from their input-output examples that generalizes to unseen programs. JEPIA learns embeddings of inputs and outputs that are rotations or reflections of each other. The paper evaluates JEPIA by examining the geometry of the learned representations, finding that the embeddings learned by modular arithmetic have a ring structure, and by evaluating the generalization of JEPIA to programs of different shapes, finding that JEPIA generalizes to program compositions of different lengths.

Critical strengths
  • JEPIA is an interesting and (to the best of my knowledge) novel approach to program induction
  • The length-generalization results are an encouraging indication that the inductive bias imposed by JEPIA’s geometric structure allows good handling of program composition
  • The evaluation seems technically correct
Critical weaknesses
  • Evaluations are only ran on programs with structure that matches the inductive bias assumed by JEPIA (invertible groups); the paper would be stronger with evaluation of how the approach generalizes to these other types of programs (even if poorly).
  • The paper would also be stronger with more motivation for the program induction setting, though this does not impact my score.
Questions for authors

How well does JEPIA generalize to programs that aren’t implementations of invertible groups, or other more complex programs?

Reviewer_Rg46OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces JEPIA, a framework for program induction from a few input-output examples, without explicitly recovering the underlying program. Inspired by JEPA, the method embeds input-output pairs into a latent space and fits an affine transformation between them. The representation is learned through losses on the transformed embeddings of novel inputs, analogous to learning representations that facilitate few-shot regression. Experiments demonstrate the approach on modular addition and Rubik’s Cube transformations.

Critical strengths

The idea of learning a representation in which program induction reduces to fitting a simple transformation is interesting and potentially useful.

Critical weaknesses
  1. Insufficient discussion of related work

Several relevant research directions are missing from the discussion.

First, mechanistic interpretability work on grokking and modular arithmetic (Nanda et al.) demonstrates that neural networks can learn structured representations of modular operations through gradient descent. This provides useful context for interpreting the learned representations in JEPIA.

More directly, several works learn representations through downstream optimization or fitting procedures. For example, ProGRIP (Unsupervised Learning of Shape Programs with Repeatable Implicit Parts) learns representations for 3D program inference using matching-based optimization, while End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scan learns features through downstream geometric alignment. Although the applications differ, these works share the principle of learning representations that facilitate a subsequent structured fitting operation.

The paper would benefit from situating its contribution relative to this broader class of methods.

  1. Limited experiments and highly constrained program families

The experiments consider narrowly defined families of transformations, making it difficult to assess the generality of the approach.

For example, the modular arithmetic experiments train and evaluate within the same mod-10 family. It is not particularly surprising that a representation optimized for this family eventually develops structure that facilitates fitting transformations from the same family.

More informative experiments would investigate:

  • Scaling: Does the approach work for larger moduli, such as n=1000?
  • Generalization: Can a single model handle varying moduli and offsets, e.g., y=(x+a)\bmod n, with n\in[5,15]?
  • Sample efficiency: Why are multiple input-output examples necessary for these transformations? Would a single pair suffice?
  • Expressiveness: Can the approach handle more complex programs, including compositions, conditionals, or branching computations?

The paper would also benefit from comparisons or discussion relative to language-model-based approaches to compositional reasoning, such as Faith and Fate: Limits of Transformers on Compositionality (Dziri et al.). This would help contextualize the capabilities and limitations of the proposed representation.

  1. Unclear conceptual contribution

The paper does not clearly articulate its central takeaway beyond demonstrating that certain program families admit latent representations in which their transformations can be fitted using affine maps.

While this is an interesting observation, the broader insight remains unclear. Under what conditions should we expect such representations to exist or be learnable? What makes this approach preferable to explicitly learning or inferring the underlying transformations? Will we always have to engineer transformations dependent on the tasks (i.e. on what does will a simple affine matrix multiplication on embeddings not suffice?).

Without a clearer characterization of its applicability, it is difficult to assess whether JEPIA introduces a generalizable principle for program induction or demonstrates an effective representation for a few carefully selected transformation families.

Questions for authors
  1. Applicability: What properties must a program family satisfy for its transformations to be represented through affine fitting in a learned latent space? How would the approach handle branching programs or nonlinear mathematical expressions?

  2. Central insight: What is the main generalizable insight the authors intend readers to take away from this work, beyond the feasibility of learning affine-compatible representations for the evaluated tasks?

2 comments

I thank the authors for their rebuttal, and the other reviewers for their reviews.

The authors have partially addressed some of my issues. I believe a revision of the related works would address my first issue. Regarding the second and third issue, I am not fully convinced. That being said, under the newer framing of this work - "a principled learning-and-inference framework for inducing unseen programs within a family by learning how the underlying data should be represented and organized" I believe its a valuable contribution (though it would require revision of the paper to align the introduction with this take-home message).

Based on this discussion, I will increase my rating to a weak accept.

Thank you for positive response. In the final version, we will revise our introduction to align with the message.

Reviewer_2igvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes JEPIA, an episodic program-induction architecture that learns a shared encoder so that an operator fitted to context input–output embeddings predicts held-out output embeddings. Its concrete instance uses an orthogonal Procrustes fit, differentiates through the fit, applies SIGReg to discourage collapse, and trains an auxiliary decoder without decoder gradients updating the encoder. Experiments study held-out additions modulo ten and unseen permutations induced by 2×2×2 cube moves. The addition embeddings develop an interpretable cyclic geometry, and the cube experiment reports approximately 75% facelet accuracy across generating lengths 1–20 after training on 19 two-move transformations. The contribution is learning a representation for reusable context-conditioned operator fitting; the cube evaluation does not explicitly execute a learned sequence of primitive operators.

Critical strengths
  1. Training the encoder through a context-fitted operator directly couples representation geometry to the inference procedure used at evaluation. The ordered-ring visualization in modular addition makes the proposed mechanism concrete. Although differentiable closed-form inner solvers have prior precedent in Bertinetto et al, already cited by the paper, the transformation-based formulation is a useful program-induction instantiation.

  2. Cube transformations are deduplicated by their induced facelet permutations, including across evaluation lengths, avoiding a superficial split of equivalent sequence strings. The modular-addition partition/seed study and extensive LPN search diagnostics also provide more evidence than a single favorable comparison.

Critical weaknesses
  1. The principal neural comparison is LPN on modular addition, while the cube result has no comparator and uses one training run. Fixed/random embeddings with the same fitting rule, a learned linear-regression inner solver, and a training-free cube baseline that matches input/output facelet color signatures across context examples would help determine what the encoder contributes. With 64 context pairs, the latter is a particularly relevant control. Approximately 75% facelet accuracy, without exact whole-cube accuracy, also leaves the reliability of complete output prediction unclear. Consequently, the evidence does not yet establish that learned geometry is needed for the reported transfer.

  2. Appendices C.2/C.4 sample inputs independently with replacement from only ten digits. A training-free baseline that stores context pairs and guesses uniformly for an unseen input has expected accuracy 1 − 0.9^(m+1), where m is the context size: 61.3% for m = 8 and 96.9% for m = 32. The latter exceeds the reported 78.1% in the one-training-addend setting of Figure 7. This is an analytical baseline, not a reproduced experiment. The five-context result of 72.4% does exceed this baseline’s 46.9%, so overlap does not explain all success. Nevertheless, pooled accuracy conflates reuse of an observed pair with inference on an unobserved input, and comparison only with 10% chance overstates the evidential value of the larger-context results.

Questions for authors
  1. Evaluation clarification: Can you report modular-addition accuracy separately on target inputs present versus absent in the context, together with the lookup baseline above? For context sizes below ten, a distinct-input, context–target-disjoint evaluation would directly test the claimed extrapolation. This could materially change my assessment of the central result.

  2. Evaluation request: How do fixed/raw and random representations with the same operator fit, and context-based facelet-permutation matching on cubes, perform under the same sampling protocol? Please also provide exact whole-state cube accuracy and variation across training seeds. These controls would establish the benefit attributable to learning the representation.

  3. Implementation clarification: Appendix B.2 notes nonuniqueness and unstable SVD differentiation for rank-deficient contexts. What stabilization or gradient rule is actually used, and how often do such contexts arise with five-dimensional embeddings and repeated digit inputs? This matters for reproducibility of the stated training algorithm.

Comments on the paper

1 comment

Dear reviewers,

Thank you for your work reviewing this paper!

The authors have submitted their responses. If you haven't already, please acknowledge them in a comment and adjust your reviews as needed. The discussion period finishes on November 18.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.