acceptodds
Under review as a conference paper at ICLR 2027

n-CLU: A Factorized Latent Dynamics Model for Deletable Compositional Generation

Abstract

A world model trusted with unseen tasks needs to go beyond generalizable predictions. Among other capabilities, dynamic tasks demand that (a) the model outputs are valid under the task's judge (b) the model composes new combinations of factors from the ones it has seen, and (c) the model can forget specific factors when the environment changes permanently. Valid, compositional generation is the primary focus of this work with machine unlearning as a supplementary goal. We present the naive-Causal Learning Unit (n-CLU), and compare against naive baselines (an MLP, an LSTM, a transformer and a continuous-input MLP) on two simulated evaluation tasks: shape-rendering and maze-solving. The n-CLU is designed as a set of factorized, principal subspaces acting on the latent space of a frozen neural-network encoder, allowing composition of encoder-extracted features. The n-CLU is then read by a request-indexed address book, and finally rendered by a frozen decoder. Deleting a training row only downdates the statistic and refits the address book, so the trained n-CLU after deletion equals an n-CLU fitted without those rows. On a held-out set of the Shapes3D dataset, the n-CLU composes new pairs and combinations of every factor. Against matched baselines trained on the same frozen encoder and decoder, the n-CLU produces more distinct, valid novel tuples per held-out combination than the strongest baseline, and beats every baseline on every evaluation set of an -way combination task across multiple seeds. To test compositional generation in a harder, world-model aligned task, we created a maze-based benchmark (CMAZE-T) with composable factors and validity rules an agent must follow during navigation, based on the state of the local environment. We also release this small-scale benchmark for valid, compositional generation and plan to expand on it in future works. On CMAZE-T the n-CLU consistently matches the strongest baseline (transformer) with a marginal seed-dependent lead, and we show both quantitative and qualitative results for the same. We finally motivate a Physics-designed extension to the n-CLU, and how it could provide clues towards designing better latent-dynamics based world models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
compositional generation, machine unlearning, memory-augmented neural networks, energy-based models, disentangled representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GmxiOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper proposes n-CLU, a factorized latent-space model designed for compositional generation of unseen factor combinations and training data deletion. It partitions latent representations extracted by a frozen encoder into multiple regions, constructing local subspaces using means and principal component directions. A tensor-product address book retrieves latent representations based on requested factor combinations, which are then rendered by a frozen decoder. By maintaining and updating local statistics, the model can approximately recover the state of a model refitted without the deleted training samples.

Experiments show that n-CLU substantially outperforms MLP, LSTM, and Transformer baselines on static attribute composition tasks in Shapes3D, while achieving comparable or slightly better performance than Transformer on single-step state transition prediction in CMAZE-T. Additional experiments validate its local data deletion capability.

Critical strengths
  1. Meaningful research problem. Compositional generalization and machine unlearning are both important research topics. Exploring model architectures that support both capabilities has potential value.
  2. Promising performance on simple compositional tasks. n-CLU consistently outperforms the baselines on multi-factor compositional generation in Shapes3D, suggesting that explicit factorized latent representations may improve compositional generalization.
Critical weaknesses

W1. Insufficient comparison with competitive structured methods.

The main paper primarily compares n-CLU with general-purpose architectures such as MLP, LSTM, and Transformer. The authors position n-CLU as a fundamental computational unit to justify this limited comparison. However, n-CLU incorporates strong structural inductive biases, and its general applicability as a computational primitive has yet to be demonstrated. Therefore, this positioning does not adequately justify the choice of baselines.

The paper lacks comprehensive comparisons with advanced structured compositional generation methods, particularly those incorporating similar factorized inductive biases. Consequently, the experiments do not sufficiently establish the unique advantages of n-CLU over alternative structured approaches.

Although additional comparisons with LACE and PoE are provided in the appendix, the authors acknowledge that these comparisons are not strictly resource-matched and that the specific configurations may not fully represent the capabilities of these methods. Thus, the current comparisons provide limited evidence for the claimed superiority of n-CLU.

W2. Limited evidence of scalability and generalization to complex tasks.

Although n-CLU substantially outperforms the baselines on Shapes3D, its performance on the more complex CMAZE-T task is only comparable to or slightly better than Transformer, with small and seed-dependent improvements.

The most convincing results are concentrated on static, low-dimensional compositional generation tasks with explicitly defined attributes. The advantage becomes much less pronounced when tasks involve rule execution order and dynamic state transitions, which are particularly relevant to world modeling.

Therefore, the current experiments provide insufficient evidence that the proposed method can generalize to more complex compositional reasoning and dynamic environments.

W3. Insufficient theoretical justification and ablation studies for the core design.

The authors construct local latent subspaces using means and PCA principal directions but provide limited justification for why this particular design is suitable for compositional generalization.

PCA captures directions of maximum variance, but its objective does not inherently guarantee semantic disentanglement or compositionality. The paper lacks a systematic analysis of this design choice and comparisons with alternative subspace construction methods.

Additional ablation studies on latent-space partitioning, subspace decomposition, and representation construction would help clarify the contributions of individual components and justify the proposed architecture.

Furthermore, despite the name Causal Learning Unit, the paper does not clearly demonstrate mechanisms for causal structure learning or causal reasoning. The CMAZE-T experiments, which involve rule-based compositional reasoning, also show no substantial advantage over the strongest baseline, leaving the causal interpretation insufficiently supported.

W4. Computational overhead undermines the practical value of the deletion mechanism.

The authors emphasize that n-CLU enables structured data deletion through statistical downdates. However, Appendix B.1 explicitly reports that refitting the address book takes longer than retraining any of the baselines from scratch.

This finding raises serious concerns about the practical efficiency of the proposed deletion mechanism. Although statistical downdates themselves may be efficient, the complete deletion procedure also involves updating subspaces and refitting the address book. Its overall efficiency therefore cannot be assessed solely from the cost of statistical updates.

Moreover, the paper lacks a clear computational complexity analysis. The authors should report the time complexity and measured runtime of individual memory writes, reads, and deletions, and analyze how these costs scale with latent dimensionality, the number of subspaces, and the training set size.

W5. Potential reliability issues with the external evaluator.

The Shapes3D experiments rely on an external classifier to determine whether generated images satisfy all requested attributes, but the reliability of this evaluation procedure is insufficiently examined.

For example, the generated images in the first two columns of the fourth row of Figure 3 appear visually similar, yet receive different validity judgments. The authors should clarify which attributes cause this discrepancy and whether the distinction is visually meaningful.

In particular, classification errors or tolerance thresholds for continuous attributes such as scale and orientation may significantly affect the final validity scores. The visual ambiguity of orientation for rotationally symmetric objects, such as spheres, also deserves attention.

Since the strong Shapes3D results constitute the primary experimental evidence for the proposed method, the authors should report the evaluator’s accuracy, per-attribute errors, and decision criteria to establish the reliability of these conclusions.

W6. Mismatch between the paper’s positioning and its experimental evidence.

The authors position n-CLU as a new fundamental computational unit and repeatedly emphasize concepts such as Latent Dynamics Model, World Model, Causal Learning Unit, and physics-inspired dynamics. However, the proposed method primarily constructs local latent subspaces from statistical quantities and retrieves compositional representations based on requests, without performing nontrivial latent dynamics.

Experimentally, Shapes3D focuses on static image attribute composition, while CMAZE-T is reduced to single-step rule-based prediction. The paper lacks key evaluations of long-horizon prediction, continuous dynamics modeling, and model scaling.

Consequently, while the results provide some evidence for structured latent compositional generation, they do not adequately support the broader claims regarding general-purpose computational primitives or latent-dynamics world models.

The authors should clarify the intended scope of the contribution and provide experimental evidence aligned with their central claims.

W7. Poor figure and table presentation.

Several figures and tables, particularly Tables 1–2 and Figure 5, are overly dense and difficult to interpret. The lack of a clear architectural diagram further hinders understanding of the proposed method.

Questions for authors

Q1. Baseline comparisons: Could the authors include comparisons with more competitive structured compositional generation methods that incorporate similar factorized inductive biases? How can the authors demonstrate that the performance gains of n-CLU arise from its specific design rather than a close alignment between its structural inductive biases and the evaluation tasks?

Q2. Methodological justification: Why are local means and PCA principal directions chosen to construct the latent representations? What theoretical or empirical evidence supports this choice over alternative subspace decomposition or structured representation methods?

Q3. Computational efficiency: Could the authors provide the computational complexity of individual memory writes, reads, and deletions, along with measured runtimes across different model scales? In particular, given that Appendix B.1 reports longer address book refitting times than baseline retraining, what are the practical advantages of the proposed deletion mechanism?

Q4. Evaluation reliability: Why do the generated images in the first two columns of the fourth row of Figure 3 receive different validity judgments? Could the authors provide the corresponding per-attribute predictions, along with the external evaluator’s accuracy and tolerance thresholds? How are visually ambiguous attributes, such as the orientation of spheres, handled during evaluation?

Q5. Paper positioning: Which specific components of n-CLU implement causal learning or latent dynamics modeling? Given the absence of nontrivial dynamical evolution and long-horizon prediction experiments, how do the authors justify positioning n-CLU as a Latent Dynamics Model and a general-purpose computational primitive?

5 comments

Thank you very much for finding our problem and work meaningful and for your targeted questions. We have used these to significantly improve the clarity of our paper, specifically the method and its positioning. We look forward to your thoughts and reply!

  • The revised manuscript:

    • The title and the opening of Sec. 1 now position the n-CLU as a factorized latent unit, not a latent dynamics model or a general-purpose primitive; the claims are scoped to what is measured. We still motivate how the n-CLU is exactly the CHLU with the dynamics turned off, and the CHLU was indeed published as a general purpose generative unit.
    • The design ablations updated in App. G: gates learned by GD instead of CD (G.1), the number of atoms per well (G.2), the atom allocation across wells (G.3), the two-stage fit (G.4) and a learned well partition (G.5).
    • Measured write, read and deletion times on one machine are in App. B.1 (Tab. 5), next to the retrain times of every baseline (Tab. 6).
    • The judge's accuracy on real images, the cross-judge agreement and the bin rule are now in App. F (Tab. 8); the rows you pointed at (now Fig. 4) carry the judge's verdict per panel as well for clarification.
    • Sec. 2 is rewritten as components, forward pass, objective and deletion around a new architecture diagram (Fig. 1); Tabs. 1–2 are reduced to mean ± sd over seeds; the maze figure is split into a main version without controls (Fig. 6) and the full one in App. E (Fig. 12).

    All of these changes have improved the readability thanks to your comments and we look forward to your thoughts on the new version!

W1 / Q1. "Insufficient comparison with competitive structured methods … how can the authors demonstrate that the gains arise from design rather than alignment between inductive biases and the evaluation tasks?"

  • LACE and PoE are scored beside the learned baselines in App.A.
    • The n-CLU clearly outperforms both significantly on all 109 held out combinations in terms of the valid tuples generated (see table below).
    • LACE and PoE were not added to the initial table since they perform close to the random control baselines despite being 3M+ parameter models (PoE is a 3M model of 6 experts) according to their own published settings, while all the learned baselines and the n-CLU (matched at 130k parameters) clearly beat both LACE and PoE. So it does not make much sense to add much larger models that perform so poorly on these datasets.
    • This is why they are not added to the main table since they are both much larger and much worse so not worth comparing against on these specific datasets.
  • Design versus bias, by control (App. F.1, Tab. 9; Tab. 1):
    • The n-CLU clearly beats both designed models as well as the random controls that test whether the encoder does most of the work.
read head (same features, decoder, judge)seed 0seed 1seed 2
n-CLU47.547.449.1
LACE0.170.070.23
PoE0.010.000.00
ridge, one-hot request9.68.213.0
n-CLU, atoms zeroed / shuffled within well / dealt across wells / random atoms0.000.000.00

W2. "Limited evidence of scalability and generalization to complex tasks."

  • We agree that this work specifically does not show generalization capability at scale, however this is the first introduction of the n-CLU and this is also the reason we have chosen to include "naive" in the nomenclature. This work motivates our future work on a physics settling based combination of the CHLU and the n-CLU and we leave that scaling to future work as part of our scoping.
  • CMAZE-T is another novelty we present that is currently single-step transition prediction. Our future work will target exapnding this dataset into a long-horizon task and a much larger dataset to evaluate on.
  • For compositional generation overall the n-CLU beats the transformer and other learned and control baselines and this is our contribution that motivates further research into the n-CLU and its Latent-Dynamics family of models.

W3 / Q2. "Why are local means and PCA principal directions chosen … evidence over alternative subspace decomposition methods? … ablation studies on partitioning, subspace decomposition, representation construction. … the paper does not clearly demonstrate mechanisms for causal structure learning."

  • For clarity we have now added an architecture diagram and re-written the methods section in the more traditional style: components of the unit, forward pass, objective and learning mechanism and inference/read stage.
  • The per-well basis from a retained statistic (means, second moment etc.) is what makes deletion possible and exact; at the deployed size the fit features lie in the per-well span (energy outside 1.4e-15), so it is lossless; the composition itself comes from the tensor-product address, which the control in the table above isolates.
  • We don't claim that principal directions give semantic disentanglement; the claims are composition under the external judge and exact deletion.
  • Ablations in the revised App. G: the gates learned by gradient descent give the same score as the contrastive-divergence unit on three seeds (Tab. 10); the maximum atoms per well is the hyperparameter and the cut gate learns how many to keep; a learned allocation of atoms across wells at half and a quarter of the directions beats a uniform one by 2.4 tuples for each of the 109 held-out combinations (Tab. 12); App. G.4 states when the two-stage fit (principal directions, then least-squares book) is the joint minimiser, and shows the full-rank case holds to float precision; App. G.5 shows that a learner of the wells themselves does not reach a partition, so it is not just the principle components but the full unit that does the unseen combinations.
  • On the name: the algorithm is effectively the CHLU by Jawahar et.al. with the dynamics settling turned off, so it keeps the family name; future work that turns the latent dynamics on to effectively combine the n-CLU and the CHLU will target compositional generation with latent dynamics in the same family. No causal inference or causal structure learning is to be inferred; "causal" is the relativistic sense of the CHLU's kinetic term, which is zero here, but we chose to keep the name given we would like to credit the CHLU for our inspiration. If the reviewer holds that the term "causal" should not be used with the relativistic term switched off, we can read n-CLU as "naive-Compositional Learning Unit". Please let us know if this change is required to improve readability.

W4 / Q3. "Computational overhead … complexity … measured runtime of writes, reads and deletions … what are the practical advantages of the proposed deletion mechanism?"

  • Complexity (App. B.1): write O(N D² / K); downdate of m rows O(m D² / K + D³ / K²), independent of N, reads no retained rows; read independent of N; address-book refit linear in N.
  • We measured on one Apple-silicon node at K = 32, D = 2,048, N = 131,072, deleting 1 % of the items (Tab. 5):
operationtime
downdate0.02 s
statistic and atoms refit1.15 s
address-book refit84 s
retrain the other baselines (MLP, transformer etc.) from scratch200 to 950 s
  • The exact part of deletion is about a second against any retrain. The address-book refit is a least-squares refit comparable to retraining one baseline: shorter than the learned baselines (MLP, LSTM, Transformer, Continuous MLP) at their respective convergence steps.
  • Runtimes at other D, K and N were not measured, but are not required since the results are presented at the measured settings; we leave further compute scaling and optimization to future work.

W5 / Q4. "Reliability of the external evaluator … Figure 3 row 4 columns 1–2 … per-attribute errors, tolerances, sphere orientation."

  • The Judge (App. F, Tab. 8a) is a separate classifier; balanced accuracy 1.000 on every factor on real images (the images behind the test requests and 4,096 uniform images.
  • Rule: exact bin on all six factors, no tolerance (10 bins per hue, 8 scale bins, 4 shapes, 15 orientation classes); every judge error found is on a decoded render. So even if a generated image looks almost close, but misses the exact values by a single bin, it is judged as incorrect. This further emphasizes that when the n-CLU produces more valid tuples than any baseline, it is exactly correct for all those tuples per combination.
  • The two cells of the submitted Fig. 3 (now Fig. 4): the baseline image fails on scale alone, one bin above the request (0.893 against 0.821); the n-CLU image beside it sits on the same boundary. The distinction is one bin, applied identically to every model.
  • Close bins of scale and orientation can look alike in a render, and that is how the benchmark is designed: the quantitative metric is the count in the tables; the images are qualitative reference (App. F). Better quantitative metrics are future work for improving the benchmarks.

W6 / Q5. "Which components implement causal learning or latent dynamics modeling? … how do the authors justify positioning n-CLU as a Latent Dynamics Model and a general-purpose computational primitive?"

  • The latent dynamics of the family are the dynamics of latent points inside the factorised latent space, not of the input system; the n-CLU is the member with those dynamics turned off, by measurement on the deployed units (App. D: shows that the n-CLU is exactly the CHLU with settling dynamics turned off. This is a design choice since this is the first application of the CHLU family to compositional generation. This work and our results motivate the further scaling of the n-CLU to have the latent dynamics features included in our future iterations to ideally further improve performance.
  • The causal term in the CHLU work that inspired our n-CLU connects to the relativistic causality term they include, which we turn off in our first naive version presented here. We kept the naming to give due credit to the CHLU, while removing the Hamiltonian to symbolize the dynamics being turned off. However, if you think this is confusing, we can change the n-CLU to mean "naive-Compositional Learning Unit" to stay aligned to the paper's scope while still keeping the nomenclature close to the CHLU family. Please let us know if you prefer us to make this change.
  • The title now reads "A Factorized Latent Unit for Deletable Compositional Generation", and Sec. 1.2 states the scope: compositional generation, refusal and deletion on Shapes3D and single-step transition prediction on CMAZE-T, a broader set of tasks left to future work.
  • Corrected in App. D: the full-rank statement and the coupling constant.

W7. "Tables 1–2 and Figure 5 are overly dense … lack of a clear architectural diagram."

  • An architecture diagram is added (Fig. 1).
  • Tables 1, 2 are now updated to show only the means and not all individual seeds making them more readable.
  • The transitions figure in the main text (Fig. 6) now shows the models without the control columns so it is more readable and the full version is Fig. 12 in App. E.
Reviewer_8B1ROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an approach, the naive-Causal Learning Unit (n-CLU), that simultaneously allows for compositional generation and machine unlearning. The goal is to propose a model that can generate valid outputs, compose new combinations of factors not observed during training and forget specific factors without re-training.

Critical strengths

The paper tries to propose a method that allows for compositional generation and machine unlearning simultaneously.

Critical weaknesses
  • The paper lacks context and background. The proposed method builds on a previous work (the Causal Hamiltonian Learning Unit (CHLU)), but the framework, task, architecture, assumptions, formalization, etc. is never introduced. Therefore, it is difficult to understand the method proposed by the authors and assess their contributions.
  • The paper does not describe the method, therefore it is extremely difficult to understand their proposed approach. They do not formalize the problem, the assumptions, the task, the model nor the architecture.
  • Since the paper lacks the formalization and description of the framework and the method, it is difficult to understand the experimental evaluation. Additionally, the experimental setup is not properly described, so in general is difficult to follow.
  • In general, the paper is difficult to follow given the lack of description of the framework, the method and the experimental setting.
Questions for authors
  • How are the wells (regions or partitions of the latent space) obtained? What are the atoms and how are they obtained? How does the number of atoms (an hyperparameter) affect the model performance?
  • Why is it necessary and what does it mean intuitively to obtain an orthonormal basis per well? Why can’t we obtain the orthonormal basis of the full latent space?
  • Reading lines 120-126 I understand that one image is generated from an specific partition (well). How is this partition selected in generation (or the ‘read’ step)?
  • Could the authors clarify how is the address book defined and trained? How is it possible to generate by using only labels, how is this indexing defined?
  • Could the authors describe the architecture? Why are the different components described in the ‘The memory footprint’ section needed? How are they trained?
  • In ‘The memory footprint’ section, the authors mention that there is a coupling and hole table used for refusal, can they clarify what do they mean by refusal? What process are they talking about? Why is it needed?
  • Can the authors clarify how is the model trained? What is the intuition behind the ‘hold-out draw’ and why is it necessary for training?
  • Does the model have any sort of stochasticity in the generative process? Or for the same request will always generate the same image?
  • Since I am struggling to understand the architecture and the proposed method itself, I cannot see what enables the model to delete factors. What makes this possible with the proposed approach, and why could the same goal not be achieved with existing methods?
  • The experimental setup is not properly introduced. Why is the method compared against MLP, LSTM, transformer and continuous-input MLP? Can the authors be more explicit on the experimental setting and the rationale behind the experiments? I do not understand for which task are these models trained in the (frozen) latent space, and how are they used in this setting (i.e., what do they output?).
  • In the experiments, I observe that the requests contain continuous (i.e., floor hue, wall hue, scale, etc) and categorical (i.e., sphere) factors, how are the categorical variables handled by the model if the latent space is continuous?
  • Can the authors clarify the CMAZE-T experimental setting? What’s the task? Generating the images? Predicting the actions? Is there any temporal modeling?
  • What’s the difference between n-CLU and n-CLU REL?
5 comments

Thank you very much for your targeted questions on the method. We agree that the previous version was too brief and thanks to your questions we were able to re-write the methods section to address each of your points that was not clear earlier, along with an architecture diagram now included. We look forward to your thoughts and reply!

  • The revised manuscript:
    • Sec. 2 (Methods) defines:
      • the components (2.1)
      • the forward pass (2.2)
      • the objective and learning (2.3)
      • deletion (2.4)
      • a new architecture diagram (Fig. 1).
  • We have also clarified deatils of the experiments where necessary in Secs. 3 and 4.
  • We have also modified the title to align better, thanks to your question on dynamics.

The n-CLU unit (reference for every answer below)

  • Between a frozen encoder and a frozen decoder, the n-CLU maps a factor-request, to a latent vector output. There are 3 modes as shown in the new architecture diagram:
    • "fit" : training
    • "read" : inference
    • "deletion" : deleting a previously learned combination and refusing to generate the deleted factor for every future request that asks for it.
      • Say I never want to generate a red cube after being trained on red cubes, we delete it and the next time a request asks for a red cube, the n-CLU emits a refusal token; at small scale this sounds like a lookup table, but we are refusing factor combinations and not factors themselves (i.e. I can still generate a red cylinder or a green cube, but just never a red cube because we deleted that combination) - which we show the lookup table baseline cannot do at scale.
  • Six components of the n-CLU: wells (latent-dim groups, designed), retained statistic and atoms (derived from the fit latents), address book (learned coefficients, least squares), gates (learned by contrastive divergence from a random start; we also added a gradient descent using the same MSE objective and find identical performance), hole table (flags for refusal).
  • Forward pass: request → hole table → address book → per-well coefficients → gates → per-well render in the atoms' basis → concatenation → decoder.
  • Please see Fig. 1 for the architecure diagram - we would appreciate feedback on how to further improve it!

Q1. "How are the wells obtained? What are the atoms and how are they obtained? How does the number of atoms affect the model performance?"

  • Wells: the 2,048 latent dimensions produced by the encoder are grouped without labels into 32 wells of a max of 64 (hyperparam) by a balanced correlation partition. So a well is a group of latent dimensions/channels, not a region of inputs; "K regions" in Sec.2 is updated to reflect this.
  • Atoms: each well's mean and leading principal directions, derived from its count, sum and second moment.
  • The maximum number of atoms per well is a hyperparameter (62 at K* = 32); the cut gate learns via contrastive-divergence how many of them to keep and the performance changes based on the number the gate learns to keep for the specific task (App. G.2). Changing the max. no. of atoms (which is the hyperparam) on shapes3D has a small efect because the gates learn a different factorization and number of atoms to keep - however on CMAZE-T the performance degrades on decreasing the max. allowed wells (App. G).
  • So the wells are analogous to the effective rank of the lantent space for a loose intuitive comparison. An analytical correlation between the two will be drawn in future work with larger scale experiments.

Q2. "Why is it necessary … to obtain an orthonormal basis per well? Why can't we obtain the orthonormal basis of the full latent space?"

  • Intuitively: If I only ever see a red cube and a green cylinder during training, and at inference I want to produce a green cube, the encoder trained on just reconstruction loss is unable to produce this unless you introduce variational sampling (in which case you can only smoothly approximate, i.e. maybe you generate a lighter shade of red and not exactly generate the requested factor). So the latent algorithm (the n-CLU or the competing baselines) all given the same latent for red cube and green cylinder must learn to disentangle color-shape (since they are semantically orthogonal, i.e., there is no effect of shape on a color or vice versa) in order to produce a green cube.
  • In a more formal sense (since the example has a lot of caveats): The wells are essentially parts of the latent vectors produced by the encoder; orthogonalizing these is an important factorization step that allows the downstream contrastive-divergence to learn the weighted combinations of atoms (defined per well) that correspond to a factor (eg. shape, color, etc), and this then allows combining factors that were never seen as a combination at training but only as individual components in different combinations.
  • So does a single well learn a single factor? it is possible if the weights learn to represent a factor with a single atom, but in practice, different combinations of atoms from different wells are learned to corresposnd to one factor.
  • The per-well vector basis is what makes deletion possible and exact: each well's atoms are re-derived from that well's own downdated statistic (Sec. 2.4).
  • A single basis of the whole latent has no per-well statistic to downdate and no block for the address book to address.

Q3. "How is this partition selected in generation (the 'read' step)?"

  • No well is selected. The address book returns learned (contrastive divergence on the MSE of the actual and n-CLU modified latents) coefficients for all 32 wells at once; each well renders its block in its atoms' basis and the decoder renders the concatenation (Sec. 2.2).

Q4. "How is the address book defined and trained? How is it possible to generate by using only labels?"

  • One learned embedding per factor value, an elementwise product over the requested factors, one learned linear map per well to the well's coefficients; fit by ridge least squares on the training rows after the gates are learned (Sec. 2.3).
  • Labels enter the unit only here. A held-out combination gets coefficients because each factor's embedding was learned from rows where that factor appeared and the product composes them (i.e. you learn what red is by looking at a red cube, a red cylinder etc.; you learn what a sphere is by looking at a yellow sphere, green sphere etc.; then you learn how to generate a red sphere - a combination you never saw).

Q5. "Could the authors describe the architecture? Why are the components in 'The memory footprint' needed? How are they trained?"

  • Architecture: Fig. 1 and Sec. 2.1.
  • The four byte counts are (1) the atoms, (2) the covariance and count statistic, (3) the address book, (4) the coupling and hole table.
  • Training (Sec. 2.3): one pass accumulates the statistic and derives the atoms; the gates are learned from a random start by persistent contrastive divergence on the latent reconstruction energy; the address book is then refit by ridge least squares on the gated read.
  • Prompted by this question we also added a gradient-descent learning regime for the gates instead of CD on the latent MSE, and we achieve identical performance with the same (added now to App.G.1.).

Q6. "What do they mean by refusal? Why is it needed?"

  • Once I delete a composition that I already trained on (for eg. green cylinder), the next time I request a green cylinder, if my model still generates it, then the deletion was not exact.
  • So after deleting green cylinder, despite the fact that the model was trained on green cylinders, the model when requested to generate that exact combination post-deletion, should refuse to generate it since it was deleted.
  • In this case the n-CLU emits a refusal signal.
  • This is specially important in world model cases: for eg. I have a foundation-scale encoder trained to drive autonmously on the fast lane on highways; then a new law prohibits autonomous cars in that lane, you either need to retrain the encoder from scratch, or you can attach guard rails that disallow this; now if I have a dynamically updating set of rules to follow, just simple guardrail harness does not scale, and this is where a learning algorithm like the n-CLU outperforms all exisitng learning based baselines (including the structured compositionl generation ones like LACE and PoE as shown in App.A).

Q7. "What is the intuition behind the 'hold-out draw' and why is it necessary for training?"

  • It is not a training device. Factor combinations drawn at random are removed from the training pool and these become the held-out test set we evaluate on (Sec. 2.4). This ensures no leakage from the test set into the training set.
  • This is proved by the random control baselines when asked to generate unseen couples drop to nearly zero with seed variance. This also proves that it is not the encoder and decoder doing the heavy-lifting for generating unseen combinations.

Q8. "Does the model .. stochasticity in the generative process?"

  • None. The same request always produces the same image. We have no sampling based steps adding variability.

Q9. "What enables the model to delete factors … and why could the same goal not be achieved with existing methods?"

  • Every learned part of the unit is a function of a retained statistic: the atoms of the per-well moments, the address book of its own count and first moment. Deleting rows subtracts them from the statistics, re-derives the atoms and refits the book; the downdate reads none of the retained rows (Sec. 2.4). This makes deletion exact.
  • Exactness: after deletion the n-CLU matches an n-CLU never trained on the deleted rows, mode deviations at floating-point round-off on every scope and seed (Sec. 4.3).
  • A learned baseline like an MLP or transformer cannot delete in this sense: its only route is a retrain with zero instances of the deleted combinaton (App. B.1, Tab. 6), and it has no refusal, so a request for a deleted value still renders something given it will always output a garbage latent of the same dimensions that the decoder can work with.
  • Refusal is indeed a peripheral harness, and you could add a post-inference filter for this, but deletion cannot be achieved with such modifications.

Q10. "Why is the method compared against MLP, LSTM,..? … for which task are these models trained in the (frozen) latent space … what do they output?"

  • These four are the usual task heads on a latent prediction model, and the n-CLU acts on the same latent space (Sec. 1.2).
  • We also show results on structured algorithms with strong inductive biases for our task like LACE and PoE in App.A, but their performance was too weak to compare against, so we leave it out of the main text.
  • Same task and training for all: a factor request to a latent read on the same frozen features, rendered by the same decoder, judged by the same judge (Sec. 3).
  • Matched to the n-CLU's atoms at about 130 k parameters (Tab. 1); a lookup table of the written items is the non-learned baseline; the continuous MLP at a much larger 610k parameters is also compared against, as well as the baselines that we trained for much larger optimizer steps past first convergence as well (in case the earlier loss decreases with longer trainings), and the n-CLU beats all baselines even when the n-CLU is at lower parameters.

Q11. "How are the categorical variables handled by the model if the latent space is continuous?"

  • Both enter the address book as one-hot codes: categorical factors directly; continuous factors as interpolated one-hots over their bins (Sec. 2.2).

Q12. "Can the authors clarify the CMAZE-T experimental setting? What's the task? … Is there any temporal modeling?"

  • The task is a single state transition prediction based on a complex set of compositional rules. The latent model given the latent vector of the current step has to output a latent that the decoder then outputs the state at the next time-step (only the 4 answer tuples, not the image itself; the image is drawn for qualitative visualizations based on the answer tuple, except the words+observation case where both the tuple and the image encoding are passed to the latent model) and this has to be a legal future forecast. It is one step transitions only at the moment, future work will expand it to longer horizons.
  • The inference request is five words (agent, key, door, tint, direction) and the answer is four facts (legal, displacement, event, pocket) (Fig. 3).
  • Rules make a move legal or illegal, and some moves are non-physical (through a wall, through a closed door); the n-CLU and every baseline must return the correct, valid, legal next state.
  • "Words plus observation" adds the maze image through the encoder along with the request tuple (Sec. 4.4). No temporal modelling is required for single state transitions, but will be added when we expand to the long horizon rollout case (but even then this will be more on the encoder and decoder itself and not the latent model itself, which is what our work presents).

Q13. "The difference between n-CLU and n-CLU REL?"

  • REL adds relational input columns (for example "wall colour equals tint colour"), like covariates carrying extra information about the world (Sec. 2.4, Sec. 4.4). Every baseline receives the same columns, and the n-CLU outperforms.
Reviewer_vvbYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They study an engineered system in the middle of a pretrained encoder/decoder model, call n-CLUE. n-CLU keeps track of sums and counts of the data. They then run two experiments: composition and deletion. For the composition experiments, they hold out some combinations (e.g. a red cylinder, where red objects and cylinders are each seen separately) and then test on them. For deletion, they delete particular training examples and then check downstream effects. What they find is that n-CLU performs well on the composition test compared to multiple baselines, and that deletion impairs behavior on related inputs.

Critical strengths

I was impressed with how well n-CLU did on the composition experiments. The baselines seem quite meaningful.

The limitations discussed are pretty thorough as well, and honest.

Critical weaknesses
  1. It's unclear how much of the behavior is due to the pretrained encoder/decoder. I realize that the baselines used in this work also use the same pretrained model, but it seems possible that the pretrained model has already factorized the data, in which case an additive model would compose well almost for free (if my intuition is correct), and some of these results might not be attributable to n-CLU. I think a simple linear baseline would test this. Similarly, what are the encoder and decoder details, and were they trained on data that includes the held-out combinations?

  2. I was also a bit confused by the world model framing, and how n-CLU relates to world models in practice given it has no dynamics.

  3. This isn't critical but just for the authors' knowledge: on line 75, it should be "Deletion is another useful feature…"

Questions for authors

Please see the questions in my Critical Weaknesses section 1 and 2, which are clarifications.

3 comments

Thank you for your questions and for the typo. Loved your brevity and targeted questions, amid the age of LLM reveiews haha! We have only used an LLM to format the results tables in these comments, so we can have a more free-flowing conversation. Looking forward to your replies!

  • The revised manuscript:
    • Methods section fully rewritten as components, forward pass and objective around a new architecture diagram (Fig. 1, Sec. 2).
    • the linear read heads, the encoder, decoder and judge details are now in App. F.
    • two corrected sentences (Sec. 4.3 adaptation; App. D); line 75.

W1(a). "It seems possible that the pretrained model has already factorized the data … a simple linear baseline would test this."

  • We fit a simple ridge linear heads on the same frozen features, fit rows, decoder, judge and hold-out at three seeds (App. F.1, Tab. 9).
  • The linear head clearly underperforms on all the same configs:
read head (same features, decoder, judge)seed 0seed 1seed 2
ridge, one-hot request → latent9.68.213.0
n-CLU47.547.449.1
  • The four landscape controls that test if the encoder-decoder pair are doing all the work (atoms zeroed; shuffled within a well; dealt across wells; random atoms) score 0.000 on the n-CLU and on both heads, every seed (Tab. 1, Tab. 9), so the encoder and decoder alone produce no valid composition. This is due to the fact that the test compositions are held-out and the encoder-decoder have nothing that incentivizes factorization in its objective.

W1(b). "What are the encoder and decoder details, and were they trained on data that includes the held-out combinations?"

  • The encoder and decoder are an autoencoder with 2,048 latent dimensions trained to reconstruct the input only, with no factorization objective. They were never trained on the held out combinations and both are frozen before any model is fitted (App. F).
  • The judge is a separate convolutional classifier per seed (1,500 steps, 8,192 images); perfect accuracy on every factor (Tab. 8a).

W2. "I was also a bit confused by the world model framing, and how n-CLU relates to world models in practice given it has no dynamics."

  • The dynamics of a CLU are the dynamics of latent points inside its own structured latent space, not the dynamics of the input system as in the case of the original CHLU paper that we are inspired by. I believe the intention was to be able to retrieve information from the latent space in the absence of the encoder that created it (this is how we interpreted the CHLU, and based our work on). While these capabilities are part of our future work, we decided to start with the naive version first and scope this paper around it.
  • The n-CLU is the naive member of the family with those latent dynamics turned off (Sec. 1, Sec. 5); the input task can be static (Shapes3D) or a transition (CMAZE-T), with dynamics involved (future versions of CMAZE-T are planned with long horizon rollouts) and the n-CLU works in both cases in terms of compositional generation, deletion and post-deletion refusal.
  • The n-CLU therefore is not itself a world model, but rather can act on a world model encoder to improve its action-conditioning and the CMAZE-T dataset was designed to show an early version of this: Say there are rules to the world, as there are rules to the cmaze-t map; say the world model needs to predict what happens in the future given a sequence of actions. In this case the n-CLU acts as a supplemenary algorithm providing better composition of unseen rule-combinations that a general purpose world model encoder misses; at the same time the n-CLU is also able to delete factors - so in the world model case if you would like to dynamically update the map/rules to make previously valid actions to be invalid in the future, the n-CLU provides deletion and refuses to generate futures that involve this newly prohibited action.
  • But overall we agree that mentioning dynamics for the n-CLU is confusing, despite our future work on the CLU including dynamics, so the title is now changed to "A Factorized Latent Unit", and Sec. 1.2 states the claims as composition, refusal and deletion on Shapes3D and single-step transition prediction on CMAZE-T.

W3. Line 75.

  • Fixed: "Deletion is another useful property".

Thanks for the response. The linear baseline and the details on the encoder and decoder answer my main question, and it seems clear now that the composition isn't just coming from the pretrained autoencoder. The new framing also clears up my confusion about world models. I'll keep my score at a 3.

Thank you very much for your analysis that led to an important ablation that now strengthens our claim. We are happy that you think our work is meaningful and worthy of acceptance!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of updates made after reviews

Thanks to our reviewers' questions, our paper is a lot more readable to a broader audience now with the following major updates made to the manuscript. We look forward to your opinions on the changes/updates and the added results!

New results in brief:

  • The frozen encoder-decoder does not compose on its own: a ridge linear head on the same features scores 9.6 / 8.2 / 13.0 against the n-CLU's 47.5 / 47.4 / 49.1 (seeds 0 / 1 / 2); LACE, PoE and the four landscape controls are near zero.
  • Gates learned by gradient descent instead of contrastive divergence give the same score, within 0.02 on every seed.
  • The exact part of deletion costs about a second; the address-book refit 84 s; retraining a baseline from scratch 200 to 950 s.

Major changes in the revised manuscript:

  • Scoping: Title changed to "A Factorized Latent Unit for Deletable Compositional Generation"; Sec. 1.2 scopes the claims to composition, refusal and deletion on Shapes3D and single-step transition prediction on CMAZE-T. The n-CLU is positioned as the member of the CHLU family with the latent dynamics turned off, not as a world model.
  • Clarity: Sec. 2 (Methods) rewritten as components (2.1), forward pass (2.2), objective and learning (2.3) and deletion (2.4), around a new architecture diagram (Fig. 1) showing the fit, read and deletion modes.
  • Encoder, decoder and judge details, the judge's accuracy on real images and the cross-judge agreement are in App. F (Tab. 8); the linear read heads on the same features are in App. F.1 (Tab. 9).
  • Reconfirming LACE and PoE results beside the learned baselines on all 109 held-out combinations and on every rung (App. A, Tab. 3).
  • Sanity checking: Design ablations in App. G: gates learned by gradient descent instead of contrastive divergence (G.1), the number of atoms per well (G.2), the atom allocation across wells (G.3), the two-stage fit (G.4) and a learned well partition (G.5).
  • Measured write, read and deletion times on one machine (App. B.1, Tab. 5) next to the retrain times of every baseline (Tab. 6).
  • Readability: Tabs. 1–2 reduced to mean ± sd over seeds; the maze figure split into a main version without controls (Fig. 6) and the full version in App. E (Fig. 12); corrected sentences in Sec. 4.3 and App. D.

New Results:

Is the frozen encoder-decoder already doing the composition? Same features, fit rows, decoder, judge and hold-out; distinct valid novel tuples per held-out combination (App. A, App. F.1, Tab. 1):

modelseed 0seed 1seed 2
n-CLU47.547.449.1
ridge linear head, one-hot request9.68.213.0
LACE0.170.070.23
PoE0.010.000.00
n-CLU controls: atoms zeroed / shuffled within well / dealt across wells / random atoms0.000.000.00

Does the learning method matter? The same unit with the gates learned by gradient descent instead of contrastive divergence, after the address-book refit (App. G.1, Tab. 10):

seedCD (Tab. 1)GD, from random start
047.49547.477
147.40447.404
249.11949.138

What does deletion cost? One Apple-silicon node, K = 32, D = 2,048, N = 131,072, deleting 1 % of the items (App. B.1, Tab. 5):

operationtime
downdate of the statistic0.02 s
statistic and atoms refit1.15 s
address-book refit (least squares)84 s
retraining a baseline from scratch without the rows200 to 950 s

The exact part of deletion is about a second against any retrain, and after deletion the n-CLU matches a unit never fitted on the deleted rows at floating-point round-off (Sec. 4.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.