acceptodds
Under review as a conference paper at ICLR 2027

Disentanglement from Nonlinear Task Sparsity

Abstract

In this work, we study representation learning through task sparsity: learning a representation in which each task depends on only small subset of latent factors. Prior work has theoretically shown that such representations can be recovered through sparsity-aware multi-task learning. However, practical realizations have been limited to linear task heads, limiting their scalability. Extending task sparsity to nonlinear predictors is challenging because sparsity is inherently discrete and therefore difficult to optimize with gradient-based methods. To address this challenge, we propose Disentanglement from nonlinear Task Sparsity (DTS), a general sparsity-aware representation learning framework for recovering disentangled representations from nonlinear multi-task datasets. \methodname formulates task sparsity as a task-conditioned binary masking problem, imposing sparsity only through the masking operation and allowing nonlinear task heads to learn from the masked features. Empirically, we evaluate \methodname on a multi-task dataset generated following the CLEVR protocol johnson2017clevr. We show that \methodname recovers the underlying disentangled factors from nonlinear tasks and the learned representation enables strong OOD generalization through sparse downstream task learning. Finally, our analysis shows that both the masking architecture and the sparsity objective are crucial for learning disentangled representations from nonlinear tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Disentangled representation, Representation Learning, Sparsity, Multi-task learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZNiMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DTS, a multi-task method built on the idea that each task depends on only a few factors of the data, so solving every task from as few latent dimensions as possible should make the dimensions line up with those factors. Unlike earlier task-sparsity methods, it does not need linear task heads: a task-conditioned binary mask selects dimensions of a binary object-token code, and a transformer answers from what is left (Eqs. 1–4). Training adds penalties on mask size and on dependence between latent dimensions, plus image reconstruction (Eqs. 5–13).

The theory (Sec. 4, App. C) shows, for binary latents and a sufficiently rich family of sparse tasks, that minimising average sparsity recovers the true latents up to reordering and relabelling (Thm. 3). The authors present these as idealised sufficient conditions (L231–235) and state that their CLEVR tasks do not generally satisfy them, so the theorem motivates the method rather than supports the experimental results (L1159–1163).

Experiments on CLEVR-style images show two things. DTS latents match the ground-truth object properties much better than the baselines do: 98 % against 61–78 % (Table 1). And they transfer better to new counting tasks under distribution shift: 93 % against 76–89 %, using about 7 latent dimensions (Table 2).

Critical strengths

1. A workable way to use task sparsity with nonlinear heads. Casting sparsity as a task-conditioned stochastic gate with a differentiable penalty lets an arbitrary transformer head sit on top, which removes the linear-head restriction that the paper identifies in earlier realisations (L057–059), and the method trains on 1M images and 3072 tasks. The clearest evidence that it yields a compact, reusable code is Table 2: downstream heads on DTS use 7 ± 1 latent dimensions against 736 for the plain transformer, at higher test accuracy (93 % vs 89 %). If the metric questions below are resolved, the 98 % matching accuracy in Table 1 would add to this.

2. Candid scoping and complete reporting. The paper states that its identifiability theorem does not cover the experimental task bank (L231–235, L1159–1163), lists its own limitations (L477–485), includes two correlation datasets chosen because they were adversarial for sparse codes (L810–811), and gives per-task results for all methods and ablations (Tables 9–11). This let me check where the method does and does not help rather than rely on averages.

Critical weaknesses

1. The experiments do not isolate what produces the disentangled representation, although the paper's headline claim is about exactly that. The abstract (L026–027) and conclusion (L475–476) state that the masking architecture and the sparsity objective are crucial for learning disentangled representations.

  • The disentanglement result (Table 1) contains no ablation rows. The ablations (Table 3) report only downstream accuracy and ∣S∣|S|, so there is no evidence on how the sparsity or MI loss changes matching accuracy.
  • No ablation removes the masking architecture: all three ablations keep binary latents and stochastic masking, while the plain baseline changes both at once to continuous latents without a mask (Table 6). The effect of binarisation is therefore confounded with that of masking. This matters because discretising the latents has been reported to promote disentanglement on its own, without any task supervision (Hsu et al., "Disentanglement via Latent Quantization", NeurIPS 2023, arXiv:2305.18378), a line of work the paper does not cite or compare against.
  • How the Table 1 metric is computed is not specified where it matters: how continuous baseline latents are turned into the binary predictions needed for TPR/TNR, how tokens are assigned to ground-truth objects, and over how many dimensions the best match is taken (each token has 4096, L709–710). The metric also only asks whether some dimension tracks a factor, not whether the remaining dimensions are free of it. The 98 % versus 78 % gap cannot be interpreted until these are known.
  • The VAE baselines use regularisation weights of 10−510^{-5} to 10−310^{-3} (Table 6) and the PACS solver was replaced by the authors' own (L721–727); no tuning is reported for either.

2. The downstream benefit over a plain multi-task transformer is small and conditional.

  • With the sparse downstream rule DTS reaches 93 ± 2 % against 89 ± 1 % for the plain baseline (Table 2, 4 seeds). Without that rule DTS is below it, 83 ± 3 % versus 90 ± 3 % (Table 5), so best against best the margin is 3 points.
  • On the spatial-relation task, which is nonlinear, the plain baseline is ahead of DTS in 14 of the 15 datasets (Table 9; about 91 % versus 87 % on average, the latter matching Table 4, which omits the plain baseline). On the antecedent count in cube_gray, cube_red and multi3, DTS reaches 55–62 % where the plain baseline reaches 71–79 % and PACS 98–99 % (Table 9); PACS is also ahead on both linear task types (Table 4). The statement that DTS is "the only method that achieves strong OOD generalization" (L416–417) is stronger than this supports.
  • Removing the MI loss changes test accuracy from 93 ± 2 to 91 ± 3 (Table 3), which is within seed variation, yet the text says either removal "significantly hinders" accuracy (L436–438).

3. The theory does not bridge the gap to the method. It motivates the inductive bias but gives no guarantee about what DTS learns, as the authors acknowledge (L1159–1163).

  • Thm. 2 restates Assumption 2 (L1117).
  • Thm. 3 requires the learned code to live in the same 0,1d\\{0,1\\}^d as the ground truth, whereas DTS uses 4096 bits per token for 32 ground-truth values.
Questions for authors

Clarifications

  1. What is the matching accuracy (Table 1) of the three ablations, and of a model with binary latents but no task-conditioned mask? Without these, which component produces disentanglement is open.
  2. How exactly is matching accuracy computed: binarisation of the continuous latents of the plain, VAE and PACS baselines; assignment of tokens to objects; and the number of candidate dimensions per method? How many dimensions of DTS remain active after pruning (L842–843), and how many of them encode each factor?
  3. How were the baseline hyperparameters in Table 6 chosen, and how does the replaced PACS solver compare with the original on a setting where the original is stable?
  4. In App. A.4 the downstream solution is selected "with its sparsity and validation accuracy". What is the exact rule, and is the 93 % versus 89 % difference in Table 2 robust to it and to more than 4 seeds?
  5. What does Thm. 3 add over existing identifiability results for sparse linear task heads, and can any part of it be extended to a learned code with slack dimensions, which is the case DTS operates in?

Minor fixes

  • Cross-references. "Table ??" in App. A (L755). L459 says "Table 3 reports" where Table 5 is meant. L403 points to Table 2 for the description of the downstream tasks, which is in Table 8.
  • Stray characters and fragments. A stray closing quote ends the "Representation Learning" paragraph of Sec. 2 ("downstream tasks.'", L102). "hese results" (L462). "learning rules. comparing naive learning rule and …" is an unfinished sentence (L460). "( Section A.4)" (L484). The middle of the AI Use Statement is not a sentence (L490–494).
  • Spelling. "probablities" (L259), "Bernouii" in the subscript of the expectation (L276), "satsifies" (L820), "PmLR" in the reference list (L630).
  • Notation and consistency. Eq. 10 ends the language tokens at wkw_k, but the question is defined as [w1,…,wL][w_1,\dots,w_L] (L265). lk,il_{k,i} is not defined (L306). Sec. 5 has KK tokens of dimension dd (L250–253), while App. A.1 has S=11S=11 tokens of dimension F=4096F=4096 and uses d=256d=256 for the decoder width (L708–710). "Slot" (L293, L312) and "token" are used for the same thing. The image resolution is 192 × 128 at L372 and 128 × 192 at L706 and L745. Datasets have "4 to 5" tasks at L402 and "five" at L814, and the split into 25 linear and 47 nonlinear tasks (L817) does not match Table 8, where three nonlinear task types over 15 datasets give 45.
  • References. Vincent et al., 2008 is cited twice in one parenthesis (L090–091). Saunders et al., 1998 has no venue (L633–634).
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZNiM (1/3)

Thank you for your constructive feedback. We address your questions and concerns in detail below and our new manuscript reflects your suggestions. Please refer to the updated PDF for new results and revisions. We believe your suggestions have substantially improved the quality and clarity of our paper, and we would greatly appreciate it if you could reconsider your score in light of the revisions.

 

[W1-1]. The experiments do not isolate what produces the disentangled representation, although the paper's headline claim is about exactly that. The abstract (L026–027) and conclusion (L475–476) state that the masking architecture and the sparsity objective are crucial for learning disentangled representations. The disentanglement result (Table 1) contains no ablation rows. The ablations (Table 3) report only downstream accuracy, so there is no evidence on how the sparsity or MI loss changes matching accuracy. What is the matching accuracy (Table 1) of the three ablations, and of a model with binary latents but no task-conditioned mask? Without these, which component produces disentanglement is open.

Thank you for your suggestion. We show the matching accuracy of the three ablations and DTS with binary latents but without task-conditioned masking. The result shows that masking is crucial for disentanglement (as removing masking has a 13% drop on matching accuracy compared to DTS). Also, the matching accuracy result shows that both sparsity and MI loss is encouraging disentanglement.

CodeColorShapeSizeMat.PosXPosYPres.Avg
Ours100.096.3100.0100.097.397.494.398.1
w/o MI99.797.099.899.891.888.793.594.6
w/o sparsity99.999.899.999.996.095.094.997.6
w/o MI & sparsity99.299.199.599.587.793.492.094.7
binary, no mask85.998.299.399.579.079.683.885.4
Vanilla72.984.181.681.479.076.681.877.7

We added this discussion and the result in Appendix D.

 

[W1-2]. No ablation removes the masking architecture: all three ablations keep binary latents and stochastic masking, while the plain baseline changes both at once to continuous latents without a mask (Table 6). The effect of binarisation is therefore confounded with that of masking. This matters because discretising the latents has been reported to promote disentanglement on its own, without any task supervision (Hsu et al., "Disentanglement via Latent Quantization", NeurIPS 2023, arXiv:2305.18378), a line of work the paper does not cite or compare against.

We thank the reviewer for pointing out the effect of binarization itself. We ran the missing ablation, a continuous latent with the masking. It is identical to the "w/o MI" recipe except that the binary straight-through bits are replaced by the continuous code. Together with the existing rows, this separates the latent type from the masking mechanism.

CodeOOD acc.
w/o MI90.9±3.490.9 \pm 3.4
w/o MI + continuous90.9±5.890.9 \pm 5.8

The results show that in our architecture, “w/o MI + continuous” and “w/o MI” does not have a big difference in test-time accuracy, meaning that the gain does not solely come from binarization. We added the reference and comparison in Appendix D.

 

[W1-3]. How the Table 1 metric is computed is not specified where it matters: how continuous baseline latents are turned into the binary predictions needed for TPR/TNR, how tokens are assigned to ground-truth objects, and over how many dimensions the best match is taken (each token has 4096, L709–710).

Thank you for pointing out the missing technical details.

For continuous latents, each column is thresholded at 9 quantiles 2, 5, 10, 25, 50, 75, 90, 95, 98%, both polarities allowed. A continuous code therefore has 9×4096×2=73,7289 \times 4096 \times 2 = 73,728 candidates per factor bit, against 4096×2=8,1924096 \times 2 = 8,192 for a binary code. This favours the continuous baselines.

For assigning the tokens to objects, we first take the candidate whose per-scene count most often equals the ground-truth count. Then, we apply Hungarian matching to match slots to objects in each scene on the Hamming cost over these 31 candidates. We added this technical details in Appendix A.1.

AuthorsOpenReview ↗

Response to Reviewer ZNiM (2/3)

[W2]. The downstream benefit over a plain multi-task transformer is small and conditional. With the sparse downstream rule DTS reaches 93 ± 2 % against 89 ± 1 % for the plain baseline (Table 2, 4 seeds). Without that rule DTS is below it, 83 ± 3 % versus 90 ± 3 % (Table 5), so best against best the margin is 3 points. On the spatial-relation task, which is nonlinear, the plain baseline is ahead of DTS in 14 of the 15 datasets (Table 9; about 91 % versus 87 % on average, the latter matching Table 4, which omits the plain baseline). On the antecedent count in cube_gray, cube_red and multi3, DTS reaches 55–62 % where the plain baseline reaches 71–79 % and PACS 98–99 % (Table 9); PACS is also ahead on both linear task types (Table 4). The statement that DTS is "the only method that achieves strong OOD generalization" (L416–417) is stronger than this supports. Removing the MI loss changes test accuracy from 93 ± 2 to 91 ± 3 (Table 3), which is within seed variation, yet the text says either removal "significantly hinders" accuracy (L436–438).

Thank you for your detailed comments. First, we would like to clarify that our main empirical claim concerns average OOD performance across all 70 downstream tasks, rather than outperforming every baseline on every individual task. As shown in Table 2, DTS achieves the highest average test accuracy (93%), outperforming the vanilla Transformer (89%). Even when comparing against the vanilla baseline without sparse downstream learning (90%), DTS retains the highest mean accuracy. Therefore, the individual cases highlighted by the reviewer do not contradict our main empirical conclusion.

Moreover, sparsity-aware downstream learning is an intended component of our framework, rather than an additional procedure introduced solely to improve performance. Our theoretical analysis explicitly motivates the need for both a disentangled representation and a sparsity-aware downstream learning rule to achieve OOD generalization. The degradation observed without sparse downstream learning is therefore consistent with this central hypothesis. Similarly, PACS performs better on linear tasks, as expected from its linear task heads, whereas DTS provides substantial improvements on nonlinear task categories. Our goal is not to outperform specialized methods on every task category, but to enable sparse learning with expressive nonlinear predictors.

For the MI ablation, we updated the paragraph to describe it as a modest decrease in mean accuracy rather than claiming that removing MI “significantly hinders” performance.

 

[W3]. The theory does not bridge the gap to the method. It motivates the inductive bias but gives no guarantee about what DTS learns, as the authors acknowledge (L1159–1163). Thm. 2 restates Assumption 2 (L1117). Thm. 3 requires the learned code to live in the same as the ground truth, whereas DTS uses 4096 bits per token for 32 ground-truth values.

Thank you for pointing this out.

We would like to highlight that the theorem’s assumptions are sufficient but not necessary for identifiability. For example, consider three fully supported binary factors, a sufficient three-bit learned representation, and tasks uniformly sampled from all pairwise conjunctions (3 combinations * 2^2 for negation = in total 12 tasks), fija,b(z)=1[zi=a, zj=b]f_{ij}^{a,b}(z)=\mathbf{1}[z_i=a,\ z_j=b]. These tasks are jointly distinguishing but do not have a common anchor. Nevertheless, still the disentangled representation uniquely minimizes expected task support. Our theorem therefore provides a motivation for the learning principle under explicit assumptions, rather than a complete characterization of when it succeeds. Deriving weaker sufficient conditions that cover broader task families is an important direction for future work.

The CLEVR results are the empirical evidence that the proposed principle remains useful beyond the theorem’s assumptions, rather than a consequence formally guaranteed by the theorem. We made this distinction explicit in the main paper in Section 4.

AuthorsOpenReview ↗

Response to Reviewer ZNiM (3/3)

[Q2] How many dimensions of DTS remain active after pruning (L842–843), and how many of them encode each factor?

On average, 94 of 4096 dimensions remain active. For each factor, 1 to 5 of the dimension encodes the factors for DTS.

 

[Q3] How were the baseline hyperparameters in Table 6 chosen, and how does the replaced PACS solver compare with the original on a setting where the original is stable? The VAE baselines use regularisation weights of to (Table 6) and the PACS solver was replaced by the authors' own (L721–727); no tuning is reported for either.

Thank you for the question. We tune only the method-specific penalty weights, choosing the best hyperparameter with the highest test-time accuracy.

CodeSweptChosen
β\beta-VAEβ∈{10−5,10−4,10−3,10−2,10−1}\beta \in \{10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}\}β=10−5\beta = 10^{-5}
TC-VAETC weight γ∈{10−3,10−2,10−1,1}\gamma \in \{10^{-3}, 10^{-2}, 10^{-1}, 1\}, KL ∈{10−3,10−4}\in \{10^{-3}, 10^{-4}\}γ=10−3\gamma = 10^{-3}, KL 10−410^{-4}
PACSα∈{10−3,10−2,10−1}\alpha \in \{10^{-3}, 10^{-2}, 10^{-1}\}, β∈{10−3,10−2,10−1}\beta \in \{10^{-3}, 10^{-2}, 10^{-1}\}α=β=10−2\alpha = \beta = 10^{-2}
OursMI weight ∈[3⋅10−8,3⋅10−5]\in [3 \cdot 10^{-8}, 3 \cdot 10^{-5}]10−610^{-6} (gate L0L_0 2⋅10−32 \cdot 10^{-3})

For the PACS, since PACS is unstable with the CLEVR dataset, we ran both solvers (original and ours) on a synthetic benchmark that satisfies PACS's own assumptions: 8 binary factors per slot, entangled by a random linear mixing into 32 well-conditioned dimensions, over 6 slots, with count and exist tasks that are linear in the pooled factors. We report learned-threshold bitmatch against the ground-truth factors (mean ± std over 3 seeds).

Inner solverBitmatch
Original99.1 ± 0.5
Ours99.6 ± 0.1

The result shows that our solver matches the performance of the original solver, showing that the replaced PACS solver works as good as the original solver in linear tasks, while being more stable on non-linear tasks, which is advantageous for PACS.

 

[Q4] In App. A.4 the downstream solution is selected "with its sparsity and validation accuracy". What is the exact rule, and is the 93 % versus 89 % difference in Table 2 robust to it and to more than 4 seeds?

Thank you for asking this question.

For selecting the solution, we first let H be the validation accuracy of a readout that uses the full code. We select the smallest support whose validation accuracy is ≥ H − 0.5. We stated this rule explicitly in Appendix A.4.

Moreover, we trained 4 additional seeds for both codes with the released recipes and ran the identical downstream learning, giving 8 seeds in total.

DTSVanillaPaired t-test p
All 8 seeds91.4 ± 3.187.9 ± 1.90.01

With 8 seeds, the difference between DTS and vanilla transformer is statistically significant (paired t-test p = 0.010).

 

[Q5] What does Thm. 3 add over existing identifiability results for sparse linear task heads, and can any part of it be extended to a learned code with slack dimensions, which is the case DTS operates in?

Thank you for this question. Theorem 3 characterizes identifiability through the number of latent coordinates needed to represent each task, rather than through the number of nonzero coefficients in a linear predictor, extending the sparse linear-predictor results (e.g., [1]) to nonlinear task heads. Regarding slack dimensions, a restricted extension follows directly when the additional coordinates are constant. Specifically, consider a representation z=(u,c)z=(u,c), where u∈[0,1]du \in [0,1]^d is sufficient for the task family and c contains arbitrary constant coordinates. Since constant coordinates provide no additional information, minimizing task support reduces to the original problem over u. Empirically, DTS’s MI objectives address the redundancy problem and drive the slack dimensions to constant. However, theory does not show that a pairwise MI objective satisfies the fixed-dimensional theorem’s assumptions.

[1] Sébastien Lachapelle et al., Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning, ICML 2023.

 

[Minor fixes]

Thank you for finding these errors! We corrected the paper accordingly.

Reviewer_YyFdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies disentangled representation learning from task sparsity. Its central idea is that many downstream tasks depend only on a small subset of underlying factors, and a representation should therefore make those factors separable so that sparse task-specific predictors can select only what is relevant. The authors propose DTS — Disentanglement from Task Sparsity, a framework for learning disentangled representations from nonlinear multi-task datasets. DTS encodes images into binary latent object tokens, predicts a task-conditioned binary mask from the task/question, and applies this mask before a nonlinear transformer task head. By penalizing the mask activation probabilities, DTS directly optimizes expected task sparsity while still allowing expressive nonlinear prediction.

Critical strengths
  1. The paper is clearly written and easy to follow. The motivation, theory, method, and experiments are presented in a clean and well-structured way. The figures and overall narrative make the main idea easy to understand.
  2. The method design targets the key challenge well. The paper identifies the main difficulty in extending task sparsity to nonlinear predictors: direct sparsity optimization is hard. The proposed task-conditioned binary masking is a simple and effective way to make sparse factor selection compatible with nonlinear task heads.
Critical weaknesses
  1. The paper assumes each task only needs a small subset of clean latent factors, which fits CLEVR well. But real-world tasks may not be that clean: useful factors can be continuous, mixed together, noisy, or context-dependent, so task sparsity may not hold.
  2. The paper compares with VAE-style methods and PACS, but lacks more direct baselines such as nonlinear sparse gating, object-centric models, concept bottleneck models, and oracle-factor variants. This makes it hard to tell whether the gain really comes from DTS’s proposed mechanism.
Questions for authors
  1. The experiments show DTS is effective with downstream sparsity-aware learning, but they do not fully isolate why it works. Without the downstream sparsity rule, DTS gets 83±2, while Vanilla gets 90±2. Does this mean the learned DTS representation alone is not better than the vanilla representation, and that the gain mainly comes from the downstream sparse selection procedure?
  2. How does DTS perform when tasks are only weakly sparse or not cleanly aligned with ground-truth factors?
  3. Can the authors add stronger nonlinear sparse-selection baselines, or clarify how much gain comes from the masking mechanism versus the task head/model capacity?
2 comments
AuthorsOpenReview ↗

Response to Reviewer YyFD

Thank you for your constructive feedback. We address your questions and concerns in detail below and our new manuscript reflects your suggestions. Please refer to the updated PDF for new results and revisions. We believe your suggestions have substantially improved the quality and clarity of our paper, and we would greatly appreciate it if you could reconsider your score in light of the revisions.

 

[Q1] The experiments show DTS is effective with downstream sparsity-aware learning, but they do not fully isolate why it works. Without the downstream sparsity rule, DTS gets 83±2, while Vanilla gets 90±2. Does this mean the learned DTS representation alone is not better than the vanilla representation, and that the gain mainly comes from the downstream sparse selection procedure?

Thank you for the question, and that is exactly the point that we would like to make in this paper. Theorem 1 shows that representation alone is not enough for OOD generalization, we have to identify the structure of the task and decide the learning rule, and learn a representation that is associated with. DTS’s representation is specialized for the learning rule with sparsity, which does not work well together with the implicit biases of the standard learning rule without the sparsity.

 

[W1, Q2] The paper assumes each task only needs a small subset of clean latent factors, which fits CLEVR well. But real-world tasks may not be that clean: useful factors can be continuous, mixed together, noisy, or context-dependent, so task sparsity may not hold. How does DTS perform when tasks are only weakly sparse or not cleanly aligned with ground-truth factors?

Thank you for this suggestion. To evaluate DTS beyond our curated task bank, we conducted an additional experiment using 16K questions from the official CLEVR-CoGenT trainA split, rather than tasks selected under our custom sparsity restriction (we use the same image data, only change the question). These programs have much less controlled sparsity: a median of 5 attribute literals per question, and the sparse relations are biased (50% involve spatial relations, 85% involve both shape and color). Then we tested the model in CLEVR-CoGenT valB split. We run DTS, Vanilla and beta-VAE for this experiment.

CodeMatching Accuracy
DTS97.6 ± 1.3
β\beta-VAE91.8 ± 1.9
Vanilla88.7 ± 2.2

DTS achieves substantially higher matching accuracy compared to the dense continuous and β\beta-VAE baseline. While this experiment still uses synthetic data and tasks, these results show additional evidence that DTS’s disentanglement extends beyond the restricted original task bank.

However, we couldn’t run the downstream learning protocol of the paper, as we train the downstream head per-task, and the valB split contains 15k questions. Designing a scalable sparsity-aware downstream learning head from the disentangled representation will be a promising future direction. We added this discussion and experiment in Section 6.3.

 

[W2, Q3] The paper compares with VAE-style methods and PACS, but lacks more direct baselines such as nonlinear sparse gating, object-centric models, concept bottleneck models, and oracle-factor variants. This makes it hard to tell whether the gain really comes from DTS’s proposed mechanism. Can the authors add stronger nonlinear sparse-selection baselines, or clarify how much gain comes from the masking mechanism versus the task head/model capacity?

Thank you for your suggestion. We additionally evaluate with top-k gating [1] (nonlinear sparse gating), Slot attention [2] (object-centric models). We did not compare with concept bottleneck models, and oracle-factor variants, as those methods assume access to ground-truth disentangled factors, while our focus is to discover those from multi-task dataset.

MethodTest Accuracy
DTS92.6 ± 2.6
Hard top-k gate (k = 16)71.1 ± 3.0
Slot Attention47.0 ± 19.4

DTS achieves higher mean test accuracy than both additional baselines. These results show that our proposed making mechanism is important, and just using a nonlinear task head is not enough. We have added these results to Appendix D.

[1] William Fedus et al., Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, JMLR 2022

[2] Francesco Locatello, Object-Centric Learning with Slot Attention, NeurIPS 2020

Thanks for your efforts and I will maintain my positive scores.

Reviewer_eyLCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies disentangled representation learning through task sparsity. The main idea is that each task should depend only on a small subset of latent factors. Prior work in this direction has mostly relied on linear task heads. The proposed DTS framework extends this principle to nonlinear predictors through task-conditioned stochastic binary masks. The model also includes a pairwise mutual-information penalty to discourage redundant latent dimensions. The paper provides theoretical results connecting task sparsity, identifiability, and OOD generalization. The empirical evaluation uses a large CLEVR-style dataset with more than 3000 auxiliary tasks and 1M images. DTS substantially improves factor recovery over the considered baselines and achieves better OOD generalization on downstream tasks affected by spurious correlations.

Critical strengths

S1. The extension of task-sparsity based disentanglement to nonlinear predictors is a meaningful contribution. This is, in my view, the strongest part of the paper. The masking formulation is simple, but it addresses a real limitation of previous approaches based on sparse linear heads.

S2. The empirical results are strong and consistent within the proposed setting. DTS reaches 98% average matching accuracy, compared with 78% for the best reported baseline. It also obtains 93% OOD test accuracy with only 7 active latent dimensions on average, compared with much denser representations for the alternatives.

I may add that the ablations also support the importance of both the sparsity objective and the mutual-information term. Overall, the empirical story is quite coherent.

Critical weaknesses

W1. The experimental setting is highly controlled and closely aligned with the proposed inductive bias. All main experiments rely on synthetic CLEVR-style scenes with explicitly known attributes and tasks that are constructed to depend on a small number of these attributes. The task bank itself is restricted so that each task depends on at most three attributes.

This makes the setting suitable for studying the proposed mechanism, but it also makes it difficult to assess how general the method is. I think the claim would be substantially stronger with at least one less controlled experiment where the relevant factors are not predefined and the sparse task structure is not explicitly engineered.

W2. The practical downstream procedure is considerably less clean than the high-level formulation suggests. The sparse downstream gate is reported to be sensitive to local optima. The authors address this by training 128 readouts and selecting the best solution according to validation accuracy and sparsity. This is not a minor implementation detail. It indicates that the sparse adaptation procedure itself is fragile. It also raises questions about computational cost and about how much performance would deteriorate with a more standard single-run optimization. A related point is that the theoretical result should be interpreted carefully. The authors explicitly state that the CLEVR task distribution does not generally satisfy the assumptions of the main identifiability theorem, and therefore the theorem does not formally explain the empirical result.

Questions for authors

I think the paper contains a good idea and the results in the controlled setting are convincing. The nonlinear extension of task-sparsity based disentanglement is interesting, and the connection with OOD generalization is well motivated. My main concern is generality. The current evidence is entirely based on a synthetic setting that is specifically constructed around sparse ground-truth factors. In addition, downstream sparse adaptation requires a heavy multi-start procedure, and the main identifiability theorem does not formally apply to the experimental task distribution. I am therefore slightly below the acceptance threshold, but I could be convinced by stronger evidence outside the current synthetic setting.

Q1. How does DTS behave when the auxiliary tasks are not explicitly designed to have sparse dependencies on known factors? Even one experiment in a less controlled setting would materially affect my assessment of the generality of the method.

Q2. How much does downstream performance degrade when reducing the number of initializations from 128 to, for example, 1, 4, 16, or 32? This would clarify whether the current optimization procedure is a practical limitation or mainly a precaution.

Q3. Can the authors clarify what part of the empirical behavior they believe is actually explained by the identifiability theory? Since the CLEVR task bank does not satisfy the theorem's sufficient conditions in general, I think this distinction should be made more explicit in the main paper.

6 comments
AuthorsOpenReview ↗

Response to Reviewer eyLC (1/2)

Thank you for your constructive feedback. We address your questions and concerns in detail below and our new manuscript reflects your suggestions. Please refer to the updated PDF for new results and revisions. We believe your suggestions have substantially improved the quality and clarity of our paper, and we would greatly appreciate it if you could reconsider your score in light of the revisions.

 

[W1, Q1]. The experimental setting is highly controlled and closely aligned with the proposed inductive bias. All main experiments rely on synthetic CLEVR-style scenes with explicitly known attributes and tasks that are constructed to depend on a small number of these attributes. The task bank itself is restricted so that each task depends on at most three attributes. How does DTS behave when the auxiliary tasks are not explicitly designed to have sparse dependencies on known factors? Even one experiment in a less controlled setting would materially affect my assessment of the generality of the method.

Thank you for this suggestion. To evaluate DTS beyond our curated task bank, we conducted an additional experiment using 16K questions from the official CLEVR-CoGenT trainA split, rather than tasks selected under our custom sparsity restriction (we use the same image data, only change the question). These programs have much less controlled sparsity: a median of 5 attribute literals per question, and the sparse relations are biased (50% involve spatial relations, 85% involve both shape and color). Then we tested the model in CLEVR-CoGenT valB split. We run DTS, Vanilla and beta-VAE for this experiment.

CodeMatching Accuracy
DTS97.6 ± 1.3
β\beta-VAE91.8 ± 1.9
Vanilla88.7 ± 2.2

DTS achieves substantially higher matching accuracy compared to the dense continuous and β\beta-VAE baseline. While this experiment still uses synthetic data and tasks, these results show additional evidence that DTS’s disentanglement extends beyond the restricted original task bank.

However, we couldn’t run the downstream learning protocol of the paper, as we train the downstream head per-task, and the valB split contains 15k questions. Designing a scalable sparsity-aware downstream learning head from the disentangled representation will be a promising future direction. We added this discussion and experiment in Section 6.3.

 

[W2, Q2]. The practical downstream procedure is considerably less clean than the high-level formulation suggests. The sparse downstream gate is reported to be sensitive to local optima. The authors address this by training 128 readouts and selecting the best solution according to validation accuracy and sparsity. This is not a minor implementation detail. It indicates that the sparse adaptation procedure itself is fragile. It also raises questions about computational cost and about how much performance would deteriorate with a more standard single-run optimization. How much does downstream performance degrade when reducing the number of initializations from 128 to, for example, 1, 4, 16, or 32? This would clarify whether the current optimization procedure is a practical limitation or mainly a precaution.

Thank you for your suggestion. We evaluated the sensitivity of DTS to the number of candidate readouts used during downstream adaptation. For this experiment, we randomly sampled K=1,4,16,32 pairs from 8 lambda values and the 16 seeds of the rollout that was initially used (in total 128). We then applied the same selection rule used in the paper.

Test accuracyK=1K=4K=16K=32K=128
DTS87.0 ± 3.389.7 ± 2.691.9 ± 2.492.4 ± 2.592.5 ± 2.6

DTS’s mean downstream performance improves as the candidate budget increases and largely saturates by K=32. However, still K=32 is computationally much heavier than the standard single-run optimization (K=1) and important for better test-time performance. We added this discussion in Appendix D, and clarified this computational trade-off and emphasized it as an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer eyLC (2/2)

[W2, Q3] A related point is that the theoretical result should be interpreted carefully. The authors explicitly state that the CLEVR task distribution does not generally satisfy the assumptions of the main identifiability theorem, and therefore the theorem does not formally explain the empirical result. Can the authors clarify what part of the empirical behavior they believe is actually explained by the identifiability theory? Since the CLEVR task bank does not satisfy the theorem's sufficient conditions in general, I think this distinction should be made more explicit in the main paper.

Thank you for pointing this out. We agree that the scope of the theoretical result should be clearer.

We would like to highlight that the theorem’s assumptions are sufficient but not necessary for identifiability. For example, consider three fully supported binary factors, a sufficient three-bit learned representation, and tasks uniformly sampled from all pairwise conjunctions (3 combinations * 2^2 for negation = in total 12 tasks), fija,b(z)=1[zi=a, zj=b]f_{ij}^{a,b}(z)=\mathbf{1}[z_i=a,\ z_j=b]. These tasks are jointly distinguishing but do not have a common anchor. Nevertheless, still the disentangled representation uniquely minimizes expected task support. Our theorem therefore provides a motivation for the learning principle under explicit assumptions, rather than a complete characterization of when it succeeds. Deriving weaker sufficient conditions that cover broader task families is an important direction for future work.

The CLEVR results are the empirical evidence that the proposed principle remains useful beyond the theorem’s assumptions, rather than a consequence formally guaranteed by the theorem. We made this distinction explicit in the main paper in Section 4.

Reviewer_eyLCOpenReview ↗

Thanks for your work. A further few questions

[W1, Q1] Thanks for the CLEVR-CoGenT exp. 1) Is it possible to have a table like Tab.4 which show the performances of each task?Do you think CLEVR-Tex could be more challenging? I'm not asking for it (I should have done it earlier), but I would like to hear your honest opinion.

[W2, Q2].Thanks! The analysis of the number of downstream initializations is convincing. This confirms that the optimization procedure is definitely sensitive to initialization.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank all reviewers for their constructive feedback! We included new experimental results and revisions in the PDF following the reviewers’ suggestions. Please check out our revised paper. We highlighted the changes in the revised version.

Here, we summarize the major changes:

  • Experiment in less controlled setting (CLEVR-CoGenT) (Section 6.3)
  • Clarification on the scope of Theorem 3. (Section 4)
  • More ablations on binarization and masking, also reporting matching accuracy for ablation studies. (Appendix D)
  • More baselines (top-k selection, Slot attention). (Appendix D)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.