- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes Disentanglement by Dual-Transformer Subtraction (DTS), a sequential disentanglement framework built on a hierarchical U-Net of dual-stream Transformers. A static stream without positional encoding produces a permutation-invariant, mean-pooled representation, which is subtracted from the full stream to separate time-invariant from time-varying factors. Two additional losses support the separation: a covariance regularization loss (CovReg) and a dynamic-only reconstruction loss. The authors also provide a theoretical result (Lemma 1) showing that the static code converges to its conditional mean given the static factor at rate , where is the input sequence length. The method is evaluated on video, audio, and time-series benchmarks.
Critical strengths
-
State-of-the-art results on audio and time series. DTS achieves state-of-the-art results on both modalities, which is the main empirical evidence supporting the effectiveness of the proposed approach.
-
A new modeling direction for sequential disentanglement. Instead of the VAE/GAN formulations and standard bottlenecks used by most prior work, DTS separates static and dynamic factors through architectural subtraction between two Transformer streams. This is a distinct design choice that could open a new line of work in the field.
Critical weaknesses
-
Limited and saturated video evaluation. Lack of quality is a known limitation of many sequential disentanglement methods, as noted in [1] and acknowledged in the introduction. However, the video evaluation relies on Sprites, which is saturated (Table 1) and no longer discriminative. The paper does not evaluate on MUG, a standard benchmark in related work it cites [1, 2, 3], nor on CelebV, VoxCeleb1, and TaiChi, which [1] introduced as harder, more complex video benchmarks for both quality and disentanglement. Without these, it is unclear whether DTS also suffers from limited quality in the video modality. At minimum, MUG should replace or complement Sprites.
-
Claims that overstate the advantage over prior work. The introduction motivates DTS by the large number of loss hyperparameters in prior methods, yet DTS itself has three loss terms: more than DiffSDA [1] and the same as DBSE [2]. Section 3.4 contrasts DTS with the "five interdependent weights of prior methods (Zhu et al., 2020; Han et al., 2021; Bai et al., 2021)" while omitting these newer methods, which is partially misleading. Similarly, Section 4.2 states that DTS learns "without task-label supervision during representation learning", but all compared methods are unsupervised, so this is not a distinguishing property. A table comparing the loss terms and loss weights of DTS against all compared methods should be added, and these claims should be rephrased in the proper context.
-
Unclear significance of Lemma 1. The paper states that "the lemma justifies the static stream as a stable estimator of what is constant over the sequence." However, showing that converges to does not imply that is itself a good representation of the static factors. The authors should justify this link more concretely, either theoretically or empirically (see Questions 6–7).
[1] DiffSDA: Unsupervised Diffusion Sequential Disentanglement Across Modalities. Zisling et al., 2026. [2] Sequential Disentanglement by Extracting Static Information From A Single Sequence Element. Berman et al., 2024. [3] Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation. Naiman et al., 2023.
Questions for authors
Questions
- Can you provide a detailed architecture description, including what the modality encoders and decoders are, where the down- and up-convolutions are located (as expected in a U-Net), and how the static stream, full stream, and decoder are built? An additional figure showing the internal structure of the blocks would help.
- How compact are the static and dynamic codes? Do their shapes differ across modalities?
- Can the dynamic-only reconstruction loss cause information collapse in the full stream?
- Is the used for static swapping taken before or after the projection?
- Did you need different architectures for video, audio, and time series?
- is used without a clear definition. Can you define it and explain how it relates to the static factors of the data?
- Do you have any analysis showing that approaching yields a better static representation, i.e., that longer input sequences lead to better static representations?
- Do you have any analysis of how well the static and dynamic codes are separated at each level of the U-Net hierarchy?
Notes / Suggestions
- Can you add reconstruction and swapping examples for video and audio as qualitative results?
- The audio experiments lack quality metrics. Can you report MSE for reconstruction quality and DNSMOS for audio quality?
- DTS cannot sample new data, which is a disadvantage relative to existing generative methods. This limitation should be stated in the main paper.
- Related work on unsupervised representation learning should be added to place this work within the broader field.
- If the dual-stream Transformer is an established architecture, can you cite the relevant paper? Citations for U-Net and "Attention Is All You Need" are also missing.
- Can you state in the abstract and introduction what converges at rate ? Also, is used before it is defined; please define it as the input sequence length.
- is used in Section 3.1 without a clear definition.
- has several different definitions across the paper, sometimes within the same subsection (e.g., Section 3.5), which makes it hard to follow. In the unnumbered equation of Section 3.5, appears as a function, while elsewhere it denotes a latent representation. Should this be ?
- Tables 3–6 are not ordered correctly.