- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DTS, a multi-task method built on the idea that each task depends on only a few factors of the data, so solving every task from as few latent dimensions as possible should make the dimensions line up with those factors. Unlike earlier task-sparsity methods, it does not need linear task heads: a task-conditioned binary mask selects dimensions of a binary object-token code, and a transformer answers from what is left (Eqs. 1–4). Training adds penalties on mask size and on dependence between latent dimensions, plus image reconstruction (Eqs. 5–13).
The theory (Sec. 4, App. C) shows, for binary latents and a sufficiently rich family of sparse tasks, that minimising average sparsity recovers the true latents up to reordering and relabelling (Thm. 3). The authors present these as idealised sufficient conditions (L231–235) and state that their CLEVR tasks do not generally satisfy them, so the theorem motivates the method rather than supports the experimental results (L1159–1163).
Experiments on CLEVR-style images show two things. DTS latents match the ground-truth object properties much better than the baselines do: 98 % against 61–78 % (Table 1). And they transfer better to new counting tasks under distribution shift: 93 % against 76–89 %, using about 7 latent dimensions (Table 2).
Critical strengths
1. A workable way to use task sparsity with nonlinear heads. Casting sparsity as a task-conditioned stochastic gate with a differentiable penalty lets an arbitrary transformer head sit on top, which removes the linear-head restriction that the paper identifies in earlier realisations (L057–059), and the method trains on 1M images and 3072 tasks. The clearest evidence that it yields a compact, reusable code is Table 2: downstream heads on DTS use 7 ± 1 latent dimensions against 736 for the plain transformer, at higher test accuracy (93 % vs 89 %). If the metric questions below are resolved, the 98 % matching accuracy in Table 1 would add to this.
2. Candid scoping and complete reporting. The paper states that its identifiability theorem does not cover the experimental task bank (L231–235, L1159–1163), lists its own limitations (L477–485), includes two correlation datasets chosen because they were adversarial for sparse codes (L810–811), and gives per-task results for all methods and ablations (Tables 9–11). This let me check where the method does and does not help rather than rely on averages.
Critical weaknesses
1. The experiments do not isolate what produces the disentangled representation, although the paper's headline claim is about exactly that. The abstract (L026–027) and conclusion (L475–476) state that the masking architecture and the sparsity objective are crucial for learning disentangled representations.
- The disentanglement result (Table 1) contains no ablation rows. The ablations (Table 3) report only downstream accuracy and , so there is no evidence on how the sparsity or MI loss changes matching accuracy.
- No ablation removes the masking architecture: all three ablations keep binary latents and stochastic masking, while the plain baseline changes both at once to continuous latents without a mask (Table 6). The effect of binarisation is therefore confounded with that of masking. This matters because discretising the latents has been reported to promote disentanglement on its own, without any task supervision (Hsu et al., "Disentanglement via Latent Quantization", NeurIPS 2023, arXiv:2305.18378), a line of work the paper does not cite or compare against.
- How the Table 1 metric is computed is not specified where it matters: how continuous baseline latents are turned into the binary predictions needed for TPR/TNR, how tokens are assigned to ground-truth objects, and over how many dimensions the best match is taken (each token has 4096, L709–710). The metric also only asks whether some dimension tracks a factor, not whether the remaining dimensions are free of it. The 98 % versus 78 % gap cannot be interpreted until these are known.
- The VAE baselines use regularisation weights of to (Table 6) and the PACS solver was replaced by the authors' own (L721–727); no tuning is reported for either.
2. The downstream benefit over a plain multi-task transformer is small and conditional.
- With the sparse downstream rule DTS reaches 93 ± 2 % against 89 ± 1 % for the plain baseline (Table 2, 4 seeds). Without that rule DTS is below it, 83 ± 3 % versus 90 ± 3 % (Table 5), so best against best the margin is 3 points.
- On the spatial-relation task, which is nonlinear, the plain baseline is ahead of DTS in 14 of
the 15 datasets (Table 9; about 91 % versus 87 % on average, the latter matching Table 4, which
omits the plain baseline). On the antecedent count in
cube_gray,cube_redandmulti3, DTS reaches 55–62 % where the plain baseline reaches 71–79 % and PACS 98–99 % (Table 9); PACS is also ahead on both linear task types (Table 4). The statement that DTS is "the only method that achieves strong OOD generalization" (L416–417) is stronger than this supports. - Removing the MI loss changes test accuracy from 93 ± 2 to 91 ± 3 (Table 3), which is within seed variation, yet the text says either removal "significantly hinders" accuracy (L436–438).
3. The theory does not bridge the gap to the method. It motivates the inductive bias but gives no guarantee about what DTS learns, as the authors acknowledge (L1159–1163).
- Thm. 2 restates Assumption 2 (L1117).
- Thm. 3 requires the learned code to live in the same as the ground truth, whereas DTS uses 4096 bits per token for 32 ground-truth values.
Questions for authors
Clarifications
- What is the matching accuracy (Table 1) of the three ablations, and of a model with binary latents but no task-conditioned mask? Without these, which component produces disentanglement is open.
- How exactly is matching accuracy computed: binarisation of the continuous latents of the plain, VAE and PACS baselines; assignment of tokens to objects; and the number of candidate dimensions per method? How many dimensions of DTS remain active after pruning (L842–843), and how many of them encode each factor?
- How were the baseline hyperparameters in Table 6 chosen, and how does the replaced PACS solver compare with the original on a setting where the original is stable?
- In App. A.4 the downstream solution is selected "with its sparsity and validation accuracy". What is the exact rule, and is the 93 % versus 89 % difference in Table 2 robust to it and to more than 4 seeds?
- What does Thm. 3 add over existing identifiability results for sparse linear task heads, and can any part of it be extended to a learned code with slack dimensions, which is the case DTS operates in?
Minor fixes
- Cross-references. "Table ??" in App. A (L755). L459 says "Table 3 reports" where Table 5 is meant. L403 points to Table 2 for the description of the downstream tasks, which is in Table 8.
- Stray characters and fragments. A stray closing quote ends the "Representation Learning" paragraph of Sec. 2 ("downstream tasks.'", L102). "hese results" (L462). "learning rules. comparing naive learning rule and …" is an unfinished sentence (L460). "( Section A.4)" (L484). The middle of the AI Use Statement is not a sentence (L490–494).
- Spelling. "probablities" (L259), "Bernouii" in the subscript of the expectation (L276), "satsifies" (L820), "PmLR" in the reference list (L630).
- Notation and consistency. Eq. 10 ends the language tokens at , but the question is defined as (L265). is not defined (L306). Sec. 5 has tokens of dimension (L250–253), while App. A.1 has tokens of dimension and uses for the decoder width (L708–710). "Slot" (L293, L312) and "token" are used for the same thing. The image resolution is 192 × 128 at L372 and 128 × 192 at L706 and L745. Datasets have "4 to 5" tasks at L402 and "five" at L814, and the split into 25 linear and 47 nonlinear tasks (L817) does not match Table 8, where three nonlinear task types over 15 datasets give 45.
- References. Vincent et al., 2008 is cited twice in one parenthesis (L090–091). Saunders et al., 1998 has no venue (L633–634).
5 comments
Response to Reviewer ZNiM (2/3)
[W2]. The downstream benefit over a plain multi-task transformer is small and conditional. With the sparse downstream rule DTS reaches 93 ± 2 % against 89 ± 1 % for the plain baseline (Table 2, 4 seeds). Without that rule DTS is below it, 83 ± 3 % versus 90 ± 3 % (Table 5), so best against best the margin is 3 points. On the spatial-relation task, which is nonlinear, the plain baseline is ahead of DTS in 14 of the 15 datasets (Table 9; about 91 % versus 87 % on average, the latter matching Table 4, which omits the plain baseline). On the antecedent count in cube_gray, cube_red and multi3, DTS reaches 55–62 % where the plain baseline reaches 71–79 % and PACS 98–99 % (Table 9); PACS is also ahead on both linear task types (Table 4). The statement that DTS is "the only method that achieves strong OOD generalization" (L416–417) is stronger than this supports. Removing the MI loss changes test accuracy from 93 ± 2 to 91 ± 3 (Table 3), which is within seed variation, yet the text says either removal "significantly hinders" accuracy (L436–438).
Thank you for your detailed comments. First, we would like to clarify that our main empirical claim concerns average OOD performance across all 70 downstream tasks, rather than outperforming every baseline on every individual task. As shown in Table 2, DTS achieves the highest average test accuracy (93%), outperforming the vanilla Transformer (89%). Even when comparing against the vanilla baseline without sparse downstream learning (90%), DTS retains the highest mean accuracy. Therefore, the individual cases highlighted by the reviewer do not contradict our main empirical conclusion.
Moreover, sparsity-aware downstream learning is an intended component of our framework, rather than an additional procedure introduced solely to improve performance. Our theoretical analysis explicitly motivates the need for both a disentangled representation and a sparsity-aware downstream learning rule to achieve OOD generalization. The degradation observed without sparse downstream learning is therefore consistent with this central hypothesis. Similarly, PACS performs better on linear tasks, as expected from its linear task heads, whereas DTS provides substantial improvements on nonlinear task categories. Our goal is not to outperform specialized methods on every task category, but to enable sparse learning with expressive nonlinear predictors.
For the MI ablation, we updated the paragraph to describe it as a modest decrease in mean accuracy rather than claiming that removing MI “significantly hinders” performance.
[W3]. The theory does not bridge the gap to the method. It motivates the inductive bias but gives no guarantee about what DTS learns, as the authors acknowledge (L1159–1163). Thm. 2 restates Assumption 2 (L1117). Thm. 3 requires the learned code to live in the same as the ground truth, whereas DTS uses 4096 bits per token for 32 ground-truth values.
Thank you for pointing this out.
We would like to highlight that the theorem’s assumptions are sufficient but not necessary for identifiability. For example, consider three fully supported binary factors, a sufficient three-bit learned representation, and tasks uniformly sampled from all pairwise conjunctions (3 combinations * 2^2 for negation = in total 12 tasks), . These tasks are jointly distinguishing but do not have a common anchor. Nevertheless, still the disentangled representation uniquely minimizes expected task support. Our theorem therefore provides a motivation for the learning principle under explicit assumptions, rather than a complete characterization of when it succeeds. Deriving weaker sufficient conditions that cover broader task families is an important direction for future work.
The CLEVR results are the empirical evidence that the proposed principle remains useful beyond the theorem’s assumptions, rather than a consequence formally guaranteed by the theorem. We made this distinction explicit in the main paper in Section 4.
Response to Reviewer ZNiM (3/3)
[Q2] How many dimensions of DTS remain active after pruning (L842–843), and how many of them encode each factor?
On average, 94 of 4096 dimensions remain active. For each factor, 1 to 5 of the dimension encodes the factors for DTS.
[Q3] How were the baseline hyperparameters in Table 6 chosen, and how does the replaced PACS solver compare with the original on a setting where the original is stable? The VAE baselines use regularisation weights of to (Table 6) and the PACS solver was replaced by the authors' own (L721–727); no tuning is reported for either.
Thank you for the question. We tune only the method-specific penalty weights, choosing the best hyperparameter with the highest test-time accuracy.
| Code | Swept | Chosen |
|---|---|---|
| -VAE | ||
| TC-VAE | TC weight , KL | , KL |
| PACS | , | |
| Ours | MI weight | (gate ) |
For the PACS, since PACS is unstable with the CLEVR dataset, we ran both solvers (original and ours) on a synthetic benchmark that satisfies PACS's own assumptions: 8 binary factors per slot, entangled by a random linear mixing into 32 well-conditioned dimensions, over 6 slots, with count and exist tasks that are linear in the pooled factors. We report learned-threshold bitmatch against the ground-truth factors (mean ± std over 3 seeds).
| Inner solver | Bitmatch |
|---|---|
| Original | 99.1 ± 0.5 |
| Ours | 99.6 ± 0.1 |
The result shows that our solver matches the performance of the original solver, showing that the replaced PACS solver works as good as the original solver in linear tasks, while being more stable on non-linear tasks, which is advantageous for PACS.
[Q4] In App. A.4 the downstream solution is selected "with its sparsity and validation accuracy". What is the exact rule, and is the 93 % versus 89 % difference in Table 2 robust to it and to more than 4 seeds?
Thank you for asking this question.
For selecting the solution, we first let H be the validation accuracy of a readout that uses the full code. We select the smallest support whose validation accuracy is ≥ H − 0.5. We stated this rule explicitly in Appendix A.4.
Moreover, we trained 4 additional seeds for both codes with the released recipes and ran the identical downstream learning, giving 8 seeds in total.
| DTS | Vanilla | Paired t-test p | |
|---|---|---|---|
| All 8 seeds | 91.4 ± 3.1 | 87.9 ± 1.9 | 0.01 |
With 8 seeds, the difference between DTS and vanilla transformer is statistically significant (paired t-test p = 0.010).
[Q5] What does Thm. 3 add over existing identifiability results for sparse linear task heads, and can any part of it be extended to a learned code with slack dimensions, which is the case DTS operates in?
Thank you for this question. Theorem 3 characterizes identifiability through the number of latent coordinates needed to represent each task, rather than through the number of nonzero coefficients in a linear predictor, extending the sparse linear-predictor results (e.g., [1]) to nonlinear task heads. Regarding slack dimensions, a restricted extension follows directly when the additional coordinates are constant. Specifically, consider a representation , where is sufficient for the task family and c contains arbitrary constant coordinates. Since constant coordinates provide no additional information, minimizing task support reduces to the original problem over u. Empirically, DTS’s MI objectives address the redundancy problem and drive the slack dimensions to constant. However, theory does not show that a pairwise MI objective satisfies the fixed-dimensional theorem’s assumptions.
[1] Sébastien Lachapelle et al., Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning, ICML 2023.
[Minor fixes]
Thank you for finding these errors! We corrected the paper accordingly.
Response to Reviewer ZNiM (1/3)
Thank you for your constructive feedback. We address your questions and concerns in detail below and our new manuscript reflects your suggestions. Please refer to the updated PDF for new results and revisions. We believe your suggestions have substantially improved the quality and clarity of our paper, and we would greatly appreciate it if you could reconsider your score in light of the revisions.
[W1-1]. The experiments do not isolate what produces the disentangled representation, although the paper's headline claim is about exactly that. The abstract (L026–027) and conclusion (L475–476) state that the masking architecture and the sparsity objective are crucial for learning disentangled representations. The disentanglement result (Table 1) contains no ablation rows. The ablations (Table 3) report only downstream accuracy, so there is no evidence on how the sparsity or MI loss changes matching accuracy. What is the matching accuracy (Table 1) of the three ablations, and of a model with binary latents but no task-conditioned mask? Without these, which component produces disentanglement is open.
Thank you for your suggestion. We show the matching accuracy of the three ablations and DTS with binary latents but without task-conditioned masking. The result shows that masking is crucial for disentanglement (as removing masking has a 13% drop on matching accuracy compared to DTS). Also, the matching accuracy result shows that both sparsity and MI loss is encouraging disentanglement.
We added this discussion and the result in Appendix D.
[W1-2]. No ablation removes the masking architecture: all three ablations keep binary latents and stochastic masking, while the plain baseline changes both at once to continuous latents without a mask (Table 6). The effect of binarisation is therefore confounded with that of masking. This matters because discretising the latents has been reported to promote disentanglement on its own, without any task supervision (Hsu et al., "Disentanglement via Latent Quantization", NeurIPS 2023, arXiv:2305.18378), a line of work the paper does not cite or compare against.
We thank the reviewer for pointing out the effect of binarization itself. We ran the missing ablation, a continuous latent with the masking. It is identical to the "w/o MI" recipe except that the binary straight-through bits are replaced by the continuous code. Together with the existing rows, this separates the latent type from the masking mechanism.
The results show that in our architecture, “w/o MI + continuous” and “w/o MI” does not have a big difference in test-time accuracy, meaning that the gain does not solely come from binarization. We added the reference and comparison in Appendix D.
[W1-3]. How the Table 1 metric is computed is not specified where it matters: how continuous baseline latents are turned into the binary predictions needed for TPR/TNR, how tokens are assigned to ground-truth objects, and over how many dimensions the best match is taken (each token has 4096, L709–710).
Thank you for pointing out the missing technical details.
For continuous latents, each column is thresholded at 9 quantiles 2, 5, 10, 25, 50, 75, 90, 95, 98%, both polarities allowed. A continuous code therefore has 9×4096×2=73,728 candidates per factor bit, against 4096×2=8,192 for a binary code. This favours the continuous baselines.
For assigning the tokens to objects, we first take the candidate whose per-scene count most often equals the ground-truth count. Then, we apply Hungarian matching to match slots to objects in each scene on the Hamming cost over these 31 candidates. We added this technical details in Appendix A.1.