- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
This paper proposes n-CLU, a factorized latent-space model designed for compositional generation of unseen factor combinations and training data deletion. It partitions latent representations extracted by a frozen encoder into multiple regions, constructing local subspaces using means and principal component directions. A tensor-product address book retrieves latent representations based on requested factor combinations, which are then rendered by a frozen decoder. By maintaining and updating local statistics, the model can approximately recover the state of a model refitted without the deleted training samples.
Experiments show that n-CLU substantially outperforms MLP, LSTM, and Transformer baselines on static attribute composition tasks in Shapes3D, while achieving comparable or slightly better performance than Transformer on single-step state transition prediction in CMAZE-T. Additional experiments validate its local data deletion capability.
Critical strengths
- Meaningful research problem. Compositional generalization and machine unlearning are both important research topics. Exploring model architectures that support both capabilities has potential value.
- Promising performance on simple compositional tasks. n-CLU consistently outperforms the baselines on multi-factor compositional generation in Shapes3D, suggesting that explicit factorized latent representations may improve compositional generalization.
Critical weaknesses
W1. Insufficient comparison with competitive structured methods.
The main paper primarily compares n-CLU with general-purpose architectures such as MLP, LSTM, and Transformer. The authors position n-CLU as a fundamental computational unit to justify this limited comparison. However, n-CLU incorporates strong structural inductive biases, and its general applicability as a computational primitive has yet to be demonstrated. Therefore, this positioning does not adequately justify the choice of baselines.
The paper lacks comprehensive comparisons with advanced structured compositional generation methods, particularly those incorporating similar factorized inductive biases. Consequently, the experiments do not sufficiently establish the unique advantages of n-CLU over alternative structured approaches.
Although additional comparisons with LACE and PoE are provided in the appendix, the authors acknowledge that these comparisons are not strictly resource-matched and that the specific configurations may not fully represent the capabilities of these methods. Thus, the current comparisons provide limited evidence for the claimed superiority of n-CLU.
W2. Limited evidence of scalability and generalization to complex tasks.
Although n-CLU substantially outperforms the baselines on Shapes3D, its performance on the more complex CMAZE-T task is only comparable to or slightly better than Transformer, with small and seed-dependent improvements.
The most convincing results are concentrated on static, low-dimensional compositional generation tasks with explicitly defined attributes. The advantage becomes much less pronounced when tasks involve rule execution order and dynamic state transitions, which are particularly relevant to world modeling.
Therefore, the current experiments provide insufficient evidence that the proposed method can generalize to more complex compositional reasoning and dynamic environments.
W3. Insufficient theoretical justification and ablation studies for the core design.
The authors construct local latent subspaces using means and PCA principal directions but provide limited justification for why this particular design is suitable for compositional generalization.
PCA captures directions of maximum variance, but its objective does not inherently guarantee semantic disentanglement or compositionality. The paper lacks a systematic analysis of this design choice and comparisons with alternative subspace construction methods.
Additional ablation studies on latent-space partitioning, subspace decomposition, and representation construction would help clarify the contributions of individual components and justify the proposed architecture.
Furthermore, despite the name Causal Learning Unit, the paper does not clearly demonstrate mechanisms for causal structure learning or causal reasoning. The CMAZE-T experiments, which involve rule-based compositional reasoning, also show no substantial advantage over the strongest baseline, leaving the causal interpretation insufficiently supported.
W4. Computational overhead undermines the practical value of the deletion mechanism.
The authors emphasize that n-CLU enables structured data deletion through statistical downdates. However, Appendix B.1 explicitly reports that refitting the address book takes longer than retraining any of the baselines from scratch.
This finding raises serious concerns about the practical efficiency of the proposed deletion mechanism. Although statistical downdates themselves may be efficient, the complete deletion procedure also involves updating subspaces and refitting the address book. Its overall efficiency therefore cannot be assessed solely from the cost of statistical updates.
Moreover, the paper lacks a clear computational complexity analysis. The authors should report the time complexity and measured runtime of individual memory writes, reads, and deletions, and analyze how these costs scale with latent dimensionality, the number of subspaces, and the training set size.
W5. Potential reliability issues with the external evaluator.
The Shapes3D experiments rely on an external classifier to determine whether generated images satisfy all requested attributes, but the reliability of this evaluation procedure is insufficiently examined.
For example, the generated images in the first two columns of the fourth row of Figure 3 appear visually similar, yet receive different validity judgments. The authors should clarify which attributes cause this discrepancy and whether the distinction is visually meaningful.
In particular, classification errors or tolerance thresholds for continuous attributes such as scale and orientation may significantly affect the final validity scores. The visual ambiguity of orientation for rotationally symmetric objects, such as spheres, also deserves attention.
Since the strong Shapes3D results constitute the primary experimental evidence for the proposed method, the authors should report the evaluator’s accuracy, per-attribute errors, and decision criteria to establish the reliability of these conclusions.
W6. Mismatch between the paper’s positioning and its experimental evidence.
The authors position n-CLU as a new fundamental computational unit and repeatedly emphasize concepts such as Latent Dynamics Model, World Model, Causal Learning Unit, and physics-inspired dynamics. However, the proposed method primarily constructs local latent subspaces from statistical quantities and retrieves compositional representations based on requests, without performing nontrivial latent dynamics.
Experimentally, Shapes3D focuses on static image attribute composition, while CMAZE-T is reduced to single-step rule-based prediction. The paper lacks key evaluations of long-horizon prediction, continuous dynamics modeling, and model scaling.
Consequently, while the results provide some evidence for structured latent compositional generation, they do not adequately support the broader claims regarding general-purpose computational primitives or latent-dynamics world models.
The authors should clarify the intended scope of the contribution and provide experimental evidence aligned with their central claims.
W7. Poor figure and table presentation.
Several figures and tables, particularly Tables 1–2 and Figure 5, are overly dense and difficult to interpret. The lack of a clear architectural diagram further hinders understanding of the proposed method.
Questions for authors
Q1. Baseline comparisons: Could the authors include comparisons with more competitive structured compositional generation methods that incorporate similar factorized inductive biases? How can the authors demonstrate that the performance gains of n-CLU arise from its specific design rather than a close alignment between its structural inductive biases and the evaluation tasks?
Q2. Methodological justification: Why are local means and PCA principal directions chosen to construct the latent representations? What theoretical or empirical evidence supports this choice over alternative subspace decomposition or structured representation methods?
Q3. Computational efficiency: Could the authors provide the computational complexity of individual memory writes, reads, and deletions, along with measured runtimes across different model scales? In particular, given that Appendix B.1 reports longer address book refitting times than baseline retraining, what are the practical advantages of the proposed deletion mechanism?
Q4. Evaluation reliability: Why do the generated images in the first two columns of the fourth row of Figure 3 receive different validity judgments? Could the authors provide the corresponding per-attribute predictions, along with the external evaluator’s accuracy and tolerance thresholds? How are visually ambiguous attributes, such as the orientation of spheres, handled during evaluation?
Q5. Paper positioning: Which specific components of n-CLU implement causal learning or latent dynamics modeling? Given the absence of nontrivial dynamical evolution and long-horizon prediction experiments, how do the authors justify positioning n-CLU as a Latent Dynamics Model and a general-purpose computational primitive?
5 comments
W3 / Q2. "Why are local means and PCA principal directions chosen … evidence over alternative subspace decomposition methods? … ablation studies on partitioning, subspace decomposition, representation construction. … the paper does not clearly demonstrate mechanisms for causal structure learning."
- For clarity we have now added an architecture diagram and re-written the methods section in the more traditional style: components of the unit, forward pass, objective and learning mechanism and inference/read stage.
- The per-well basis from a retained statistic (means, second moment etc.) is what makes deletion possible and exact; at the deployed size the fit features lie in the per-well span (energy outside 1.4e-15), so it is lossless; the composition itself comes from the tensor-product address, which the control in the table above isolates.
- We don't claim that principal directions give semantic disentanglement; the claims are composition under the external judge and exact deletion.
- Ablations in the revised App. G: the gates learned by gradient descent give the same score as the contrastive-divergence unit on three seeds (Tab. 10); the maximum atoms per well is the hyperparameter and the cut gate learns how many to keep; a learned allocation of atoms across wells at half and a quarter of the directions beats a uniform one by 2.4 tuples for each of the 109 held-out combinations (Tab. 12); App. G.4 states when the two-stage fit (principal directions, then least-squares book) is the joint minimiser, and shows the full-rank case holds to float precision; App. G.5 shows that a learner of the wells themselves does not reach a partition, so it is not just the principle components but the full unit that does the unseen combinations.
- On the name: the algorithm is effectively the CHLU by Jawahar et.al. with the dynamics settling turned off, so it keeps the family name; future work that turns the latent dynamics on to effectively combine the n-CLU and the CHLU will target compositional generation with latent dynamics in the same family. No causal inference or causal structure learning is to be inferred; "causal" is the relativistic sense of the CHLU's kinetic term, which is zero here, but we chose to keep the name given we would like to credit the CHLU for our inspiration. If the reviewer holds that the term "causal" should not be used with the relativistic term switched off, we can read n-CLU as "naive-Compositional Learning Unit". Please let us know if this change is required to improve readability.
W4 / Q3. "Computational overhead … complexity … measured runtime of writes, reads and deletions … what are the practical advantages of the proposed deletion mechanism?"
- Complexity (App. B.1): write O(N D² / K); downdate of m rows O(m D² / K + D³ / K²), independent of N, reads no retained rows; read independent of N; address-book refit linear in N.
- We measured on one Apple-silicon node at K = 32, D = 2,048, N = 131,072, deleting 1 % of the items (Tab. 5):
| operation | time |
|---|---|
| downdate | 0.02 s |
| statistic and atoms refit | 1.15 s |
| address-book refit | 84 s |
| retrain the other baselines (MLP, transformer etc.) from scratch | 200 to 950 s |
- The exact part of deletion is about a second against any retrain. The address-book refit is a least-squares refit comparable to retraining one baseline: shorter than the learned baselines (MLP, LSTM, Transformer, Continuous MLP) at their respective convergence steps.
- Runtimes at other D, K and N were not measured, but are not required since the results are presented at the measured settings; we leave further compute scaling and optimization to future work.
W5 / Q4. "Reliability of the external evaluator … Figure 3 row 4 columns 1–2 … per-attribute errors, tolerances, sphere orientation."
- The Judge (App. F, Tab. 8a) is a separate classifier; balanced accuracy 1.000 on every factor on real images (the images behind the test requests and 4,096 uniform images.
- Rule: exact bin on all six factors, no tolerance (10 bins per hue, 8 scale bins, 4 shapes, 15 orientation classes); every judge error found is on a decoded render. So even if a generated image looks almost close, but misses the exact values by a single bin, it is judged as incorrect. This further emphasizes that when the n-CLU produces more valid tuples than any baseline, it is exactly correct for all those tuples per combination.
- The two cells of the submitted Fig. 3 (now Fig. 4): the baseline image fails on scale alone, one bin above the request (0.893 against 0.821); the n-CLU image beside it sits on the same boundary. The distinction is one bin, applied identically to every model.
- Close bins of scale and orientation can look alike in a render, and that is how the benchmark is designed: the quantitative metric is the count in the tables; the images are qualitative reference (App. F). Better quantitative metrics are future work for improving the benchmarks.
W6 / Q5. "Which components implement causal learning or latent dynamics modeling? … how do the authors justify positioning n-CLU as a Latent Dynamics Model and a general-purpose computational primitive?"
- The latent dynamics of the family are the dynamics of latent points inside the factorised latent space, not of the input system; the n-CLU is the member with those dynamics turned off, by measurement on the deployed units (App. D: shows that the n-CLU is exactly the CHLU with settling dynamics turned off. This is a design choice since this is the first application of the CHLU family to compositional generation. This work and our results motivate the further scaling of the n-CLU to have the latent dynamics features included in our future iterations to ideally further improve performance.
- The causal term in the CHLU work that inspired our n-CLU connects to the relativistic causality term they include, which we turn off in our first naive version presented here. We kept the naming to give due credit to the CHLU, while removing the Hamiltonian to symbolize the dynamics being turned off. However, if you think this is confusing, we can change the n-CLU to mean "naive-Compositional Learning Unit" to stay aligned to the paper's scope while still keeping the nomenclature close to the CHLU family. Please let us know if you prefer us to make this change.
- The title now reads "A Factorized Latent Unit for Deletable Compositional Generation", and Sec. 1.2 states the scope: compositional generation, refusal and deletion on Shapes3D and single-step transition prediction on CMAZE-T, a broader set of tasks left to future work.
- Corrected in App. D: the full-rank statement and the coupling constant.
W7. "Tables 1–2 and Figure 5 are overly dense … lack of a clear architectural diagram."
- An architecture diagram is added (Fig. 1).
- Tables 1, 2 are now updated to show only the means and not all individual seeds making them more readable.
- The transitions figure in the main text (Fig. 6) now shows the models without the control columns so it is more readable and the full version is Fig. 12 in App. E.
Thank you very much for finding our problem and work meaningful and for your targeted questions. We have used these to significantly improve the clarity of our paper, specifically the method and its positioning. We look forward to your thoughts and reply!
The revised manuscript:
All of these changes have improved the readability thanks to your comments and we look forward to your thoughts on the new version!
W1 / Q1. "Insufficient comparison with competitive structured methods … how can the authors demonstrate that the gains arise from design rather than alignment between inductive biases and the evaluation tasks?"
W2. "Limited evidence of scalability and generalization to complex tasks."