acceptodds
Under review as a conference paper at ICLR 2027

From Cooperation to Winner-Takes-All: Identifiable Slot Disentanglement under Max-Pooling via Annealed Log-Sum-Exp

Abstract

Unsupervised disentangled representation learning enables interpretable concept discovery in domains ranging from images to biological systems. Recent advances establish provable identifiability, but they rely on strict structural assumptions like sparsity or the linear independence of first- and second-order derivatives in additive decoders. Violating these conditions can degrade model performance in real-world settings. For example, datasets with overlapping objects or missing colour channels can cause a single representational slot to cover multiple objects. We study max-pooled decoders to account for scenarios where concepts mask one another. We also provide a formal proof of identifiability for these generative processes requiring fewer smoothness assumptions than previous work. We propose a temperature-annealed Log-Sum-Exp (LSE) pooling framework, which approaches the identifiable max pooling with better optimisation behaviour. At low inverse temperature, this function behaves as an easily optimisable additive decoder. As the inverse temperature increases, it smoothly converges to strict max-pooling, where our guarantees hold. This mechanism dynamically routes backward gradients to the specific representational slots most responsible for the reconstruction. On synthetic datasets with occlusion, our annealed LSE decoders outperform standard additive models and recover the latents better than naive max-pooling. Our theorems also cover the softmax masks of object-centric decoders in their max limit, and on grayscale CLEVR6 such decoders separate objects more reliably than a joint decoder. Finally, we demonstrate that annealing improves the alignment of latents with annotated cell types in six of eleven single-cell transcriptomics atlases.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Unsupervised Representation Learning, Identifiability, Disentanglement, Single-Cell Transcriptomics, Object-Centric Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hTzYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

First of all, I thank the authors for taking their time to write the paper. The core idea seems to be extending the additive decoder framework to incorporate cases when the data modality carries signals in the inputs that occlude one another. The authors provide occluded objects in images as a core example, but I can see how the concept can transfer across modalities (e.g. for scRNA-Seq data, having perturbations that might mask expression of genes belonging to cell identity). In this regard, I found the motivation clear.

The solution, as the authors suggest, lies in how the latent information is aggregated during decoding: one might assume averaging is a good choice, but for cases like occlusion, the object in the front of the view plane would completely obstruct signals of other objects. The authors therefore propose a family of aggregations that recapitulates averaging and max-pooling at opposite ends, controlled by the inverse temperature \tau. Importantly, their theory reportedly rests on milder assumptions (compared to previous work which require differentiable decoders and second-order conditions). The paper is also honest in that the theory provided is for \tau at infinity, and that this is an open question for future work. Alongside this, the authors also did their due diligence thoroughly in exploring most of the relevant literature (to which I have provided some suggestions below for extensions).

 

Overall, taken as a purely theoretical extension over a case in applying additive decoders, the paper seems to be a significant contribution. That being said, there are some parts of the paper that I found to be lacking, particularly for experimental applications, although I am inferring that the paper is mainly focusing on theory for that matter. In any case, I provide the authors with a list of strengths and weaknesses that can be acted upon, which I believe will make the paper overall stronger.

Critical strengths
  • The problem is clearly motivated, as to why existing theory excludes occlusion. Proposition 6 is short and general. It makes precise the informal claim that smooth decoders cannot represent visible occlusion boundaries, and it is likely to be useful independently of the rest of the paper. The authors provide a new identifiability result for decoders, with weaker smoothness assumptions, which seems particularly significant for those working on additive decoders. I also did not notice anything completely out of line with regards to proofs and assumptions during my reading (although the appendix was skimmed).

  • The design of the solution is quite intuitive and well approached. Focusing on the pooling side, and classifying the two ends as existing pieces of literature positions the paper and its contribution well.

Critical weaknesses
  • The experimental validation leaves much to be desired across all fronts. Although the authors have mainly focused on practically showing the results of their theoretical work, baselines are severely lacking. The only non-author baseline is IA for CLEVR6. Additionally, if the question is the general case of obtaining disentangled representations without any supervision, and the matter is finding a model that is good at this, I am sure there should be baselines the authors could include (\beta-VAEs, \beta TCVAEs, DIPVAE, FactorVAE etc. among others that the authors cite). Then again, the sections do establish what the theory proposed with regards to filling the gaps in the spectrum of “Cooperation - Winner Takes It All”. Similarly, for scRNA-Seq data, the experiments provided by the authors do not motivate why one should opt for annealing if the gains are as marginal as what Table 3 (similarly Tables 11-12) suggests. In my opinion, it would be much better to connect the representation quality with biological statements if that is the end goal (e.g. what would one have missed if one did not use annealed LSE)?. This further requires baselines that may not even be geared towards disentanglement (such as scVI which without batches is essentially a VAE with a Negative binomial / Poisson likelihood), or similarly adaptation of baselines listed above using a similar likelihood adaptation to compare if one gains anything by adopting annealing.

  • Overall, not really clear why one would go for the annealed members of the family, when one can choose ends of the spectrum. I am wondering if there are cases when one would prefer not being on the ends of the spectrum (i.e. full cooperation or the max regime). This would be made even stronger if this corresponds to some intuitive quantity / quality that is easy to understand (e.g. maybe there are certain scRNAseq settings where occlusion of expression occurs in a specific way etc.). If the answer to this is that the temperature is essentially only to allow max-pooling and is irrelevant to the setting, then one would definitely require comparing to baselines to see how this approach fares against others.

Questions for authors
  • For all datasets, it would be great to have baselines (doesn’t need to be exhaustive), especially for the authors’ setting where they find the current literature lacking. The synthetic experiment demonstrates this already (with the additive panel vs LSE), but it would make the experimental side a lot stronger if a similar analysis / intuition / qualitative message could be obtained for other datasets.

  • For scRNA-Seq data, cell types are the easiest properties of the cell to disentangle, as they are the dominant axis of expression. I am wondering if there’s a way to extend this experiment to see if this approach allows disentangling factors of variation that are more “interesting” compared to commonly employed representation learning methods such as scVI?

Annealed members of the family: the way to view them etc. (are they purely for approximating as \tau goes to infinity, or do the finite values have something interesting about them). I may have missed this in the paper, which in this case might mean that outlining this benefit in a more apparent way, perhaps through figures, would be better.

As the paper mainly evaluates on scRNA-seq, side information (“Employing an auxiliary variable”) is routinely used. Examples include temporal and spatial covariates structuring latent factors (MEFISTO, Velten et al., 2022; NSF, Townes & Engelhardt, 2023), paired modalities split into shared and modality-specific latents (APOLLO, Zhang et al., 2026), with identifiability results for the unpaired multi-domain case (Sturma et al., 2023). Condition labels are likewise used to separate invariant from condition-specific factors (contrastiveVI, Weinberger et al., 2023; scDisInFact, Zhang et al., 2024; DisCoVR, Slavutsky et al., 2026), with identifiability theory for this model class (Lopez et al., 2024) and sparse-mechanism-shift models for perturbation data (Lopez et al., 2023). The authors might consider using any of these references to better position the proposed pooling to clarify when a max-type inductive bias is needed and when auxiliary information already suffices.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer hTzY (part 1)

We thank Reviewer hTzY for the thorough review, the constructive suggestions, and the references. Following the review, we added baselines in all three settings and an analysis of a biological program beyond cell types (the cell cycle). We answer each point below.

The experimental validation leaves much to be desired across all fronts. ... baselines are ... that the authors cite).

We thank the reviewer for this suggestion. On the synthetic data, we added β\beta-VAE, β\beta-TCVAE, FactorVAE and DIP-VAE-II with a joint decoder, tuning each by grid search on the validation block-MIG, which uses the true latents and thus favors them (Table 1, Table 15, Appendix L.1). None reaches annealed LSE or LSE at a constant τ\tau between 1 and 8. The best, β\beta-VAE, has a mean block-MIG of 0.77 on slider and 0.52 on clock, against 0.99 and 0.85 for annealed LSE (five seeds). This is aligned with the unidentifiability of nonlinear decoders without inductive biases or auxiliary information (Locatello et al., 2019). We describe new single-cell baselines in the next response point.

Similarly, for scRNA-Seq data, the experiments provided by the authors do not motivate why ... anything by adopting annealing.

We agree that the LMS-SMI gain of annealing over τ=1\tau=1 on single-cell data is modest, since most of the improvement over mean pooling (which scores 21 to 43% lower than τ=1\tau=1 on every atlas) comes from max-like pooling itself. Annealing gives a much sharper gene-to-latent assignment than τ=1\tau=1 (thus more interpretable and closer to our identifiability guarantees) while avoiding the degradation of a constant τ=8\tau=8. Please see point 1 of our General reply to all reviewers.

To connect representation quality with biological statements, we evaluated the cell cycle and closely related cell-type pairs (Figure 10, Appendix K.3). Without max-like pooling, the cell-cycle signal is spread over several latents, and without annealing, some closely related cell types (myofibroblasts vs. pericytes in intestine, cDC1 vs. cDC2 in spleen) overlap in the matched latents. Please see point 2 of our General reply to all reviewers.

As the reviewer suggested, we also trained scVI (128 latents, same hidden dimensions) and an NB β\beta-TCVAE based on it, with three seeds on all 11 atlases. Both score below the LSE models on every atlas (mean LMS-SMI 0.400.40 and 0.540.54 vs. 0.610.61 to 0.670.67, Tables 3 and 11).

Overall, not really clear why one would go for the annealed members of the ... approach fares against others.

We use a finite τ\tau primarily to train the model toward the identifiable max end (τ→∞\tau \to \infty, Section 3.2). A small τ\tau yields less sharp feature-to-slot assignment and poorer occlusion modeling, whereas a large τ\tau is harder to optimize. Finding the best intermediate constant τ\tau requires hyperparameter tuning, as the optimal value varies across datasets and initializations (for example, τ=8\tau=8 performs well on slider but worse on clock, while τ=32\tau=32 is best on clock but degrades on slider; Figure 3b, Tables 6 and 7). With annealing, slots cooperate early at low τ\tau, and as τ\tau rises, single slots win features with growing margins (new Figure 4). Please see point 1 of our General reply to all reviewers.

We also appreciate the comment on expression in scRNA-seq data. Biological processes in cells interact extensively. Inhibition is a type of interaction in which one process slows down or completely blocks another. For example, CDKN1A protein is known to inhibit the cell cycle (e.g., to prevent tumor growth). By analogy with occlusion in imaging data, such inhibition could be modeled as masking, in which one latent variable masks the activity of another. Whether masking decoders describe such systems better than additive ones is an open question that we have not tested, and modeling and analyzing such complex systems is beyond the scope of the current work. We now mention this as an interesting future direction in the conclusion (Section 5).

For all datasets, it would be great to have baselines (doesn’t need to be ... obtained for other datasets.

We agree and added baselines on both the synthetic data (first response above) and the 11 single-cell atlases (scVI and NB β\beta-TCVAE). On single-cell data, joint-decoder VAEs and mean pooling fail to isolate individual biological programs or closely related cell types into single latents as cleanly as LSE (Tables 3, 11 and 13).

For intuition and a qualitative comparison across datasets, we added Figure 4 (slot specialization during annealing on synthetic data) and Figure 10 (Appendix K.3, latent separation of closely related cell-type pairs across models). Please also see point 2 of our General reply to all reviewers for the cell-cycle results.

AuthorsOpenReview ↗

Reply to Reviewer hTzY (part 2)

For scRNA-Seq data, cell types are the easiest properties of the cell to disentangle, ... methods such as scVI?

We agree. We additionally evaluated the cell cycle (G2M and S phase scores), a well-characterized process that varies within cycling cell types, across all 11 atlases. Please see point 2 of our General reply to all reviewers.

Annealed members of the family: the way to view them etc. (are they purely ... figures, would be better.

As the reviewer suggests, we added new Figure 4, which shows how the annealed model evolves from low to high τ\tau during training on slider. Please see point 1 of our General reply to all reviewers for an empirical comparison between annealing and constant temperatures, and our response to the third comment above for a discussion on the role of annealing.

As the paper mainly evaluates on scRNA-seq, side information ("Employing an auxiliary variable") is ... auxiliary information already suffices.

We thank the reviewer for these valuable references. We have expanded our extended literature review (Appendix J, under a new paragraph "Auxiliary and supervised information in single-cell omics") to discuss these works and position our approach in relation to them.

Briefly, as with vision datasets where supervised segmentation masks and annotations aid disentanglement, auxiliary information clearly benefits scRNA-seq analysis when reliable labels are available. Because auxiliary variables are typically incorporated through additional loss terms rather than architectural inductive biases, they can be directly combined with architectural biases such as max-mask decoders (our work). However, when the auxiliary variable is itself the target of analysis (such as the perturbation in genetic screens), models must employ it with care, as label noise or unmodeled confounders directly transfer into the latent representation. A detailed discussion is provided in Appendix J.

References:

[1] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Scholkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. Proceedings of the 36th International Conference on Machine Learning, 2019.

Reviewer_fzCJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a simple fix to the problem of object-centric models not always segmenting properly. Their insight is that, much like when we use Gumbel-Softmax distributions, it is beneficial to slowly increase the level of competition that slots engage in when decoding. They achieve this by annealing the softmax temperature throughout training. They provide both theoretical and experimental evidence that this is a good strategy. Specifically, their prove several properties related to the behavior of decoders depending on whether they have soft or hard ownership of input features (i.e. pixels). They then show that annealed competition between slots leads to better segmentation across several datasets.

Critical strengths

The main strengths of the paper are the fact that they explore a fairly intuitive idea, and that they explore the implications of this idea and related design choices both theoretically and experimentally. The experiments are fairly extensive, covering both visual segmentation in standard vision datasets and medical imaging.

Critical weaknesses

The main weakness is that the paper can be really hard to read at times. There is way too much focus on providing theorems which, while useful, can be really hard to follow. Sometimes notation is not even introduced in the theorem until it is referenced (what is h in Theorem 9?). I would focus on the intuitions and leave the formalizations to the appendix. There is also very little illustration of the points and datasets used, specially the circle and slider which are new datasets and have are only introduced with two examples. Similarly, issues with the decoder could be described via diagram (the easiest example is proposition 6), which would make the points clearer.

I can go on but my overall point is that the presentation needs to be polished a lot.

Questions for authors

Can you include better diagrams that describe the issues with the decoders and illustrate your point better?

Furthermore, you have many more diagrams and plots in the appendix which could illustrate your points and results much better. Those should be front and center.

It seems that the raw scores on the Clevr dataset are not much better for the annealed LSE, yet the plots in the appendix show the opposite. What is happening here?

Finally, is the model the is annealed used with hardmax at test time? It wasn't clear to me from the text but seems like the logical step.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer fzCJ

We thank Reviewer fzCJ for the careful reading and concrete suggestions on presentation, which we adopted: the revision puts intuition before formal statements and adds two new figures to the main text.

The main weakness is that the paper can be really hard to read at ... h in Theorem 9?).

We agree that intuitive explanations will further improve clarity. We now present the intuitions before the formal statements, define core concepts such as the mixing map and disentanglement in words, and present the assumptions with greater intuition. In addition, the local and global theorems are summarized in one paragraph, and the identifiability result is stated informally in Section 3.1, with all formal statements in the appendix.

We also clarify that the identifiability theorem for hard-max and max-mask decoders (which allow modeling of occlusion) is a core contribution of our work. The revision now explains it more intuitively, together with annealing as the practical optimization procedure that approaches this theoretical limit.

Regarding hh in Theorem 9 (now Theorem 28), although hh was introduced in Definition 1 of the original submission (now Definition 5), we agree that a reader missing the context will be confused without its role ("the mixing map hh") repeated three pages later. Now, every theorem and lemma states the role of each symbol, and all notation is collected in Appendix B.

I would focus on the intuitions and leave the formalizations to the appendix. There ... introduced with two examples.

We thank the reviewer for this concrete suggestion. Section 2 now defines the core concepts informally (slots, true latents, the ground-truth and learned decoders, the mixing map) and illustrates them on slider (Figure 2a, defined in Section 4.1), deferring formal definitions to Appendix C. Section 3.1 gives the assumptions and results informally, each followed by its intuition (formal statements: Definitions 3 to 10, Theorems 14, 24 and 28).

For intuition on the datasets and results:

  • New Figure 2 illustrates our setup on slider and clock, including a disentangled box and how different pooling functions combine occluding shapes.
  • New Figure 4 shows the effect of annealing during training on slider.
  • New Figure 10 (Appendix K.3) plots the matched latents of three pairs of closely related cell types.
  • Figure 3(a) shows two held-out slider scenes and Figure 7 (Appendix K.1) five held-out slider and clock scenes, with the true and learned ownerships.

Similarly, issues with the decoder could be described via diagram (the easiest example is ... illustrate your point better?

Thank you for this suggestion. New Figure 5 illustrates the switching result (Proposition 25, formerly Proposition 6). It plots the slot outputs and the feature value along one direction of the true latent, for a max-mask decoder and for the hard max, and marks the switches that each assumption rules out. Section 3.1 now explains the issue in words first.

Furthermore, you have many more diagrams and plots in the appendix which could illustrate ... be front and center.

We agree. Within the page limit, new baselines moved into the main tables (Tables 1 and 3), and Figures 2 and 4 are new. Figure 9 (Appendix K.2) keeps the complete grayscale CLEVR6 comparison, as it mainly shows that IA does not train stably on grayscale.

It seems that the raw scores on the Clevr dataset are not much better ... What is happening here?

Yes, the scores of the softmax-mask decoders with a constant τ\tau and with annealing are similar in Table 2. The reason is that with unconstrained mask logits, the network can rescale them during training to cancel τ\tau without affecting reconstruction, so both reach similar mask sharpness. We now standardize the mask logits across slots before temperature multiplication and the softmax (called STD) to limit this cancellation and study the annealing effect. Please see point 3 of our General reply to all reviewers.

We checked Figure 5 (now Figure 9) in the appendix and found that its two unconstrained softmax-mask rows were labelled "LSE", which can be mistaken for the LSE pooling of slot colours followed by an MLP, an ablation on the necessity of masking that appears only in Appendix K.2.1 (Table 9, formerly Table 10). The labels are now fixed to "softmax mask". The figure shows that IA (the interaction asymmetry model) on grayscale merges objects in its worst seed, consistent with Table 2. We added rows for the new STD runs and show the worst, median and best seeds for them.

Finally, is the model the is annealed used with hardmax at test time? It ... like the logical step.

We thank the reviewer for this interesting question. In short, no. All reported results use the final training temperature, but a hard maximum at test time changes reconstruction very little for the annealed models (on CLEVR6, those with standardized logits). Please see point 4 of our General reply to all reviewers.

Reviewer_1GXROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when a model can recover separate objects or factors from observations where their contributions overlap. It proves identifiability for decoders that select one slot's contribution per feature, including through a separate mask. To make training easier, it proposes gradually sharpening a smooth pooling operation toward a maximum. Experiments cover synthetic images, object separation on CLEVR6, and alignment with cell-type annotations in single-cell data.

Critical strengths
  • The paper has a clear overall structure, and the examples help explain the method.
  • The writing is mostly clear and specific.
  • The motivation is reasonable: when objects overlap, one object can hide another rather than simply add to its appearance. Studying decoders that allow this is useful for understanding object representations.
  • The synthetic experiments show that an identifiable model can still be difficult to train. Smooth pooling often recovers the factors better than hard-max pooling, giving a concrete reason to study the training procedure.
Critical weaknesses
  • The theorem covers a restricted form of occlusion. Assumption A3 requires each object to retain enough visible information to recover all its latent variables locally. Fully hidden objects, or attributes whose only visible evidence is hidden, fall outside this assumption.
  • Annealing does not necessarily make the trained model behave more like a maximum. The network can shrink its outputs as the inverse temperature increases.
  • The CLEVR6 results do not show a clear advantage from annealing. These results mainly support decoding each slot separately and combining the outputs through a mask.
  • Better alignment with cell-type labels does not establish that the model separates different biological processes. The matching score can improve even if other latent dimensions still mix or duplicate information.
  • The theory also assumes exact reconstruction and an invertible mapping between true and learned representations. The experiments use approximate reconstruction and finite temperature. The paper does not establish how much of the guarantee survives these differences.
Questions for authors
  • Can you give a realistic example of partial occlusion that satisfies A3, including an object with several varying attributes?
  • How much does pooling actually sharpen during annealing in the main experiments? Does greater sharpness consistently lead to better recovery of the underlying factors?
  • Does annealing outperform fixed τ\tau = 8 on the single-cell datasets?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer 1GXR (part 1)

We thank Reviewer 1GXR for the careful and constructive review, and in particular for the precise questions on A3, sharpness, and the single-cell temperature. We answer each point below.

The theorem covers a restricted form of occlusion. Assumption A3 requires each object to ... fall outside this assumption.

We agree that A3 excludes both cases. The revision shows that the first one cannot be avoided by any method (new Appendix G.2). For a max-mask decoder where an object is completely hidden on a box of true latents, the new Proposition 26 constructs a smooth mixing map hh that mixes the hidden slot without changing the outputs (f∘h=ff \circ h = f). The learned model f∘hf \circ h thus reproduces the data exactly and meets every assumption that the true model meets (both fail A3 on that box), yet hh is not block-disentangled on an open subset of that box. A completely hidden slot therefore cannot be identified from the observations alone. For the hidden attribute case, we expect a similar counterexample to be straightforward (replace the bump function in Proposition 26 with one that takes the other latents as input and changes only the part of the latent responsible for the hidden attribute).

Nevertheless, one can restrict the domain to the region where A3 holds. If A3 fails only inside a closed ball KK with interior K∘K^\circ whose complement Z∘∖K∘Z^\circ \setminus K^\circ is connected (which makes sense, as shapes usually hide locally), one permutation π\pi of the slots still holds everywhere outside K∘K^\circ (Z∘∖K∘⊆DπZ^\circ \setminus K^\circ \subseteq D_\pi). Proposition 27 formalizes this.

Regarding the excluded region, Proposition 27 additionally shows, under some assumptions, that the values of the learned block hπ(i)h_{\pi(i)} inside K∘K^\circ fill exactly the bounded region enclosed by its values on the boundary of KK. For example, for a 1D slot hidden over zi∈(a,b)z_i \in (a, b), hπ(i)(z)h_{\pi(i)}(z) must lie strictly between its values at zi=az_i = a and zi=bz_i = b. One could also add regularizers, such as the compositional consistency objective of Wiedemer et al. (2024), to extend the learned decoder to the excluded region KK. Whether their guarantee extends to max-mask decoders remains open for future work.

We would like to add that A3 plays the role of the full-rank conditions of earlier works, such as the sufficient nonlinearity of Lachapelle et al. (2023) (Section 3.1).

Annealing does not necessarily make the trained model behave more like a maximum. The ... the inverse temperature increases.

We thank the reviewer for raising this important point. This cancellation occurs fully in softmax-mask decoders (CLEVR6) with unconstrained mask logits, where the logits can rescale without affecting reconstruction, to a limited extent on single-cell data, and not in LSE value pooling on the synthetic data, where pooling sharpens substantially with τ\tau. We now introduce standardization of mask logits (STD) to limit this in CLEVR6 models. Please see point 3 of our General reply to all reviewers.

The CLEVR6 results do not show a clear advantage from annealing. These results mainly ... outputs through a mask.

We agree. Since unconstrained softmax-mask decoders cancel τ\tau by rescaling their mask logits, we do not see a clear difference. We therefore study the advantage of annealing in STD models, which limit this rescaling. Please see point 3 of our General reply to all reviewers.

Better alignment with cell-type labels does not establish that the model separates different biological ... mix or duplicate information.

We agree. We evaluated the cell cycle, a biological process that varies within cell types, across all 11 atlases, and found that LSE pooling concentrates it into single latents consistently better than the baselines. Please see point 2 of our General reply to all reviewers.

The theory also assumes exact reconstruction and an invertible mapping between true and learned ... guarantee survives these differences.

We agree that the guarantee holds only with exact reconstruction through an invertible mixing map hh (A2, Section 3.1), and the revised conclusion (Section 5) now states this. This is a standard assumption for identifiability results. For example, additive decoders (Lachapelle et al., 2023) and interaction asymmetry (Brady et al., 2025) make the same assumption.

We also quantified the finite-temperature gap with a hard maximum at test time. Please see point 4 of our General reply to all reviewers.

AuthorsOpenReview ↗

Reply to Reviewer 1GXR (part 2)

Can you give a realistic example of partial occlusion that satisfies A3, including an object with several varying attributes?

Yes. Consider a square and a triangle of equal area, each with a varying color, rotation and position. However they move, parts of the object behind stay visible (Appendix G.2, Figure 6). Having enough variation inside the objects (e.g. shades from the center to the edges), allows the Jacobian of the visible pixels to have full rank. We think that in practice, enough of the objects of interest usually remains visible to infer what lies behind. However, this is just an intuition behind A3 and not a guarantee.

How much does pooling actually sharpen during annealing in the main experiments? Does greater ... of the underlying factors?

We thank the reviewer for this interesting question. We visualized the pooling becoming sharper during annealing on an example (new Figure 4). For more analysis, please see point 3 of our General reply to all reviewers.

Greater sharpness alone does not consistently improve recovery, since a large constant τ\tau also gives sharp pooling but may optimize poorly. Over the 105 runs per synthetic dataset, the Spearman correlation of block-MIG with the winner share pmax⁡p_{\max} is only ρ=0.13\rho = 0.13 on slider and 0.180.18 on clock (p=0.18p = 0.18 and 0.070.07), against 0.850.85 and 0.780.78 for reconstruction R2R^2. A rank regression combining R2R^2 and pmax⁡p_{\max} explains recovery best (Appendix L.4, Figure 11).

Does annealing outperform fixed τ = 8 on the single-cell datasets?

Yes. On all 11 single-cell atlases, annealed LSE (τ:0.5→8\tau: 0.5 \to 8) outperforms a fixed τ=8\tau=8, even though both end equally sharp. Please see point 1 of our General reply to all reviewers.

References:

[1] T. Wiedemer, J. Brady, A. Panfilov, A. Juhos, M. Bethge, and W. Brendel. Provable compositional generalization for object-centric learning. The Twelfth International Conference on Learning Representations, 2024.

[2] S. Lachapelle, D. Mahajan, I. Mitliagkas, and S. Lacoste-Julien. Additive decoders for latent variables identification and Cartesian-product extrapolation. Advances in Neural Information Processing Systems 36, 2023.

[3] J. Brady, J. von Kugelgen, S. Lachapelle, S. Buchholz, T. Kipf, and W. Brendel. Interaction asymmetry: a general principle for learning composable abstractions. The Thirteenth International Conference on Learning Representations, 2025.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General reply to all reviewers (part 1)

We thank all three reviewers for their thoughtful feedback and detailed technical questions. We are encouraged that the reviewers found our motivation "reasonable" (1GXR) and "clearly motivated" (hTzY), our core idea "fairly intuitive" (fzCJ), and the manuscript well structured, with "mostly clear and specific" writing (1GXR). We are especially glad that Reviewer hTzY viewed our identifiability result as "a significant contribution" that is "particularly significant for those working on additive decoders", and highlighted Proposition 6 (now Proposition 25 in Appendix G, stated informally in Section 3.1) as short, general, and useful. We are also encouraged that our empirical evidence was seen as "fairly extensive" (fzCJ) and "a concrete reason to study the training procedure" (1GXR).

Main changes in the revision:

  • We rewrote Sections 2 and 3 to present the intuition rather than formal statements (fzCJ). No concept has changed, and the formal statements are unchanged in the appendix.
  • We added new baselines for the synthetic and single-cell data (hTzY).
  • We added new models to study annealing on CLEVR6 (1GXR, fzCJ).
  • We added more biological evaluation and insight beyond cell types (1GXR, hTzY).
  • We added new theory on complete occlusion (1GXR).
  • We added more diagnostics and intuitive figures (fzCJ, hTzY).

In the revision, new or modified text is blue and relocated text dark green.

Definitions (from Appendix L.3). The margin is ln⁡(w(1)/w(2))\ln(w_{(1)}/w_{(2)}) for the two largest pooling weights, measured in nats. A margin of mm nats means the winning weight is eme^m times the second largest. The winner share pˉmax⁡\bar p_{\max} is the largest pooling weight, averaged over features.

1. Annealing versus a large constant τ (1GXR, hTzY)

Annealing approaches the identifiable hard-max limit (τ→∞\tau \to \infty, Section 3.2). At the sum end (τ→0\tau \to 0), the decoder cannot represent occlusion (Figure 2c). At the max end or a large constant τ\tau, each feature passes gradients almost only to its current winning slot, which keeps other slots from participating early in training (Figure 1, Table 1) and gives sharp pooling but poor reconstruction and disentanglement on the synthetic and CLEVR6 data (largest τ\tau in Tables 1 and 2). Annealing avoids both failures.

On the synthetic data, annealing from 0.50.5 to τ\tau outperforms a constant τ\tau at the same final value in 18 of 20 comparisons of recovery (block-MIG) and reconstruction (R2R^2) across final temperatures from 88 to 128128, while ending equally sharp (Appendix L.5, Table 20).

On the single-cell atlases, we additionally tested τ=8\tau=8. Annealing (τ:0.5→8\tau: 0.5 \to 8) outperforms a constant τ=8\tau=8 on all 11 atlases (Tables 3 and 11). Sharpness does not explain this difference, as both end comparably sharp (mean winner share 0.910.91 vs. 0.870.87, Tables 11 and 18). Against constant τ=1\tau=1, annealing improves mean LMS-SMI modestly (0.6680.668 vs. 0.6540.654, higher on 6 of 11 atlases), but gives a much sharper model (mean winner share 0.910.91 vs. 0.640.64), allowing cleaner gene-to-latent assignment for interpretability and bringing the model closer to the hard-max regime of our identifiability theorems.

2. Biological factors beyond cell types (1GXR, hTzY)

To test whether LSE pooling separates biological processes that vary within cell types, we evaluated the cell cycle (G2M and S phase scores) across all 11 atlases (Section 4.3, Appendix K.3), measuring the R2R^2 of linear fits on the single best latent (R12R^2_1) and on all latents (Rall2R^2_{\mathrm{all}}). A low R12R^2_1 means no single latent captures the phase, and a low ratio R12/Rall2R^2_1/R^2_{\mathrm{all}} means the signal is split across latents. Because not every cell type cycles, we evaluated both across all cells (Table 13) and within cycling cell types (Table 14). The table gives the means over all cells (Table 11).

Mean R12R^2_1 (share R12/Rall2R^2_1/R^2_{\mathrm{all}})Annealed LSELSE, τ=8LSE, τ=1Mean poolingscVINB β-TCVAE
G2M, all cells0.31 (61%)0.30 (61%)0.30 (58%)0.16 (32%)0.15 (36%)0.21 (45%)
S, all cells0.22 (55%)0.20 (57%)0.21 (53%)0.11 (31%)0.10 (33%)0.11 (38%)

LSE pooling (annealed and fixed τ\tau) outperforms all baselines, which spread the cell-cycle signal over several latents. These gains come primarily from max-like pooling rather than annealing, as τ=1\tau=1 is close to annealed LSE.

Figure 10 (Appendix K.3) shows what is missed without max-like pooling or annealing for three pairs of closely related cell types. Annealed LSE separates them in 7 of 8 atlas and pair combinations, whereas mean pooling and scVI mix them in all 8. Annealing also separates two pairs that τ=1\tau=1 mixes (myofibroblasts vs. pericytes in intestine, cDC1 vs. cDC2 in spleen).

Note: We now evaluate on all cells rather than 50k-cell subsamples, changing LMS-SMI by at most 0.010.01 (old numbers in Table 10).

AuthorsOpenReview ↗

General reply to all reviewers (part 2)

3. Role of annealing and output-scale cancellation (1GXR, fzCJ)

A temperature can be cancelled if the network can scale slot outputs (including presence logits) without changing the loss.

Increased sharpness in LSE value pooling. Here the pooling weights are computed from the slot outputs themselves, so the network cannot rescale them by 1/τ1/\tau without changing the pooled output and the reconstruction loss. It could still bring losing outputs closer to the winning one, but in practice it does not. On slider the margin grows almost in proportion to τ\tau, from 1.761.76 nats at τ=1\tau=1 to 89.989.9 at τ=64\tau=64 (new Table 17, Appendix L.3), and the winner share rises from 0.7540.754 at τ=1\tau=1 to 0.9950.995 for annealing (Figure 4 shows the pooling sharpening during annealing). On single-cell data, the cancellation is limited. The mean winner share rises from 0.640.64 at τ=1\tau=1 to 0.870.87 at τ=8\tau=8 and 0.910.91 for annealing, and the mean margin from 2.02.0 to 10.210.2 and 11.211.2 nats (Tables 11 and 18).

Cancellation of τ\tau in raw-logit softmax masks. Here separate mask logits determine the pooling weights, so the network can shrink them by 1/τ1/\tau to cancel τ\tau without changing the loss. This happens in the unconstrained softmax-mask models on CLEVR6, whether τ\tau is constant or annealed. They all end with a winner share of 0.920.92 to 0.970.97 and a large margin of 5.15.1 to 6.16.1 nats (Table 19). This explains why annealing shows no clear advantage there. These decoders discard τ\tau and internally approach a max-mask model, where our identifiability results hold.

Standardized logits (STD). To limit this rescaling, we standardized the mask logits across slots before the multiplication by τ\tau and the softmax (STD, updated Tables 2 and 19). With STD masks, annealing (τ:0.5→8\tau: 0.5 \to 8) matches the unconstrained models in J-ARI and gives the highest JIS of all models (how exclusively each pixel depends on one slot). A constant τ=1\tau=1 gives softer masks (winner share 0.690.69, margin ∼2.4\sim 2.4 nats), and constant τ=8\tau=8 and τ=32\tau=32 suffer from optimization instability (worst seeds in Table 2). Since STD limits the cancellation, annealing gives sharp masks without this collapse (winner share ∼0.99\sim 0.99, margin ∼7.4\sim 7.4 nats; Table 19). The STD rows of Table 2 (mean ±\pm sd over three seeds):

STD maskJ-ARI colorJIS colorJ-ARI grayscaleJIS grayscale
τ: 0.5 to 8 (annealed)98.1 ±\pm 1.098.0 ±\pm 0.498.1 ±\pm 0.198.4 ±\pm 0.0
τ=197.8 ±\pm 0.189.8 ±\pm 1.498.1 ±\pm 0.193.5 ±\pm 0.4
τ=895.0 ±\pm 4.794.7 ±\pm 1.661.7 ±\pm 53.497.2 ±\pm 2.8
τ=3238.9 ±\pm 44.396.3 ±\pm 3.565.0 ±\pm 56.197.0 ±\pm 3.0

4. Closeness to the hard maximum (1GXR, fzCJ)

For the trained annealed models, we measured the winner share pˉmax⁡\bar p_{\max}, the mean margin, and the share of features with a small margin over pixels (synthetic, CLEVR6) or genes (single-cell):

Annealed modelWinner share pˉmax⁡\bar p_{\max}Mean margin (nats)Features with margin <3<3 nats
Slider0.9950.99550.350.32.22.2%
Clock0.9960.99645.645.61.91.9%
Single-cell (11-atlas mean)0.9080.90811.211.22828%
CLEVR6, STD masks, color0.9880.9887.37.33.03.0%
CLEVR6, STD masks, grayscale0.9820.9827.57.55.05.0%

The synthetic models and CLEVR6 with STD masks are close to a hard maximum, whereas single-cell models are softer, with more genes at small margins, though still far sharper than at τ=1\tau=1 (Table 18).

All reported results use the final training temperature (now stated at the opening of Section 4). In Appendix L.2 (Table 16), we replaced the pooling of the trained models by a hard maximum at test time and subtracted the expected log⁡(k)/τ\log(k)/\tau offset of LSEτ\mathrm{LSE}_\tau. The annealed models barely change: R2R^2 by at most 0.00010.0001 on the synthetic data, and the held-out negative log-likelihood from 767.99767.99 to 768.33768.33 on single-cell data (against 762.76762.76 to 781.49781.49 at τ=1\tau=1). On CLEVR6 with STD masks, the MSE (×10−3\times10^{-3}) rises by only 0.050.05 with J-ARI preserved, against at least 0.240.24 for unconstrained softmax masks and for STD at τ=1\tau=1 (Table 16). Changing the pooling after training leaves the encoder, the latents, block-MIG and LMS-SMI unchanged.

Although the trained annealed models behave very close to the hard-max regime at test time in these cases, this is not a proof of identifiability at finite τ\tau. Proposition 29 (Appendix I.1) shows that a finite τ\tau alone does not give identifiability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.