- Insufficient discussion of related work
Several relevant research directions are missing from the discussion.
First, mechanistic interpretability work on grokking and modular arithmetic (Nanda et al.) demonstrates that neural networks can learn structured representations of modular operations through gradient descent. This provides useful context for interpreting the learned representations in JEPIA.
More directly, several works learn representations through downstream optimization or fitting procedures. For example, ProGRIP (Unsupervised Learning of Shape Programs with Repeatable Implicit Parts) learns representations for 3D program inference using matching-based optimization, while End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scan learns features through downstream geometric alignment. Although the applications differ, these works share the principle of learning representations that facilitate a subsequent structured fitting operation.
The paper would benefit from situating its contribution relative to this broader class of methods.
- Limited experiments and highly constrained program families
The experiments consider narrowly defined families of transformations, making it difficult to assess the generality of the approach.
For example, the modular arithmetic experiments train and evaluate within the same mod-10 family. It is not particularly surprising that a representation optimized for this family eventually develops structure that facilitates fitting transformations from the same family.
More informative experiments would investigate:
- Scaling: Does the approach work for larger moduli, such as n=1000?
- Generalization: Can a single model handle varying moduli and offsets, e.g., y=(x+a)\bmod n, with n\in[5,15]?
- Sample efficiency: Why are multiple input-output examples necessary for these transformations? Would a single pair suffice?
- Expressiveness: Can the approach handle more complex programs, including compositions, conditionals, or branching computations?
The paper would also benefit from comparisons or discussion relative to language-model-based approaches to compositional reasoning, such as Faith and Fate: Limits of Transformers on Compositionality (Dziri et al.). This would help contextualize the capabilities and limitations of the proposed representation.
- Unclear conceptual contribution
The paper does not clearly articulate its central takeaway beyond demonstrating that certain program families admit latent representations in which their transformations can be fitted using affine maps.
While this is an interesting observation, the broader insight remains unclear. Under what conditions should we expect such representations to exist or be learnable? What makes this approach preferable to explicitly learning or inferring the underlying transformations? Will we always have to engineer transformations dependent on the tasks (i.e. on what does will a simple affine matrix multiplication on embeddings not suffice?).
Without a clearer characterization of its applicability, it is difficult to assess whether JEPIA introduces a generalizable principle for program induction or demonstrates an effective representation for a few carefully selected transformation families.
I thank the authors for their rebuttal, and the other reviewers for their reviews.
The authors have partially addressed some of my issues. I believe a revision of the related works would address my first issue. Regarding the second and third issue, I am not fully convinced. That being said, under the newer framing of this work - "a principled learning-and-inference framework for inducing unseen programs within a family by learning how the underlying data should be represented and organized" I believe its a valuable contribution (though it would require revision of the paper to align the introduction with this take-home message).
Based on this discussion, I will increase my rating to a weak accept.